Autosubmit is a tool to create, manage and monitor experiments using computing clusters, HPC platforms and supercomputers remotely via ssh. Get involved or contact us: Autosubmit GitLab: https://earth.bsc.es/gitlab/es/autosubmit Autosubmit Mailing List: autosubmit@bsc.es HOW TO DEPLOY/SETUP AUTOSUBMIT FRAMEWORK ======================================== - Autosubmit has been tested: with the following Operating Systems: * Linux Debian on the following HPC platforms: * Ithaca (IC3 machine) * Stargate (IC3 machine) * MareNostrum (BSC machine) * MareNostrum3 (BSC machine) * HECToR (EPCC machine) * Lindgren (PDC machine) * C2A (ECMWF machine) * CCA (ECMWF machine) * ARCHER (EPCC machine) - Pre-requisites: These packages (bash, python2, sqlite3, git-scm > 1.8.2, subversion) must be available at local machine. These packages (argparse, dateutil, pyparsing, numpy, pydotplus, matplotlib, paramiko, saga-python, pythondialog, mock, portalocker) must be available for python runtime. And the machine is also able to access HPC platforms via password-less ssh. - Install Autosubmit > pip install autosubmit or download, unpack and "python setup.py install" - Create a repository for experiments: Say for example "/cfu/autosubmit" then "autosubmit configure" and follow instructions - Create a blank database: "autosubmit install" HOW TO USE AUTOSUBMIT ===================== > autosubmit -h > autosubmit expid --HPC ithaca --description "experiment is about..." Say for example, "cxxx" is 4 character based expid generated by system automatically. First character is a letter, the other three alfa-numeric characters allow to identify uniquely the experiment. > vi /cfu/autosubmit/cxxx/conf/expdef_cxxx.conf > vi /cfu/autosubmit/cxxx/conf/autosubmit_cxxx.conf > vi /cfu/autosubmit/cxxx/conf/platforms_cxxx.conf > vi /cfu/autosubmit/cxxx/conf/jobs_cxxx.conf > autosubmit create cxxx > nohup autosubmit run cxxx & Cautions: - Before launching autosubmit check the following stuff: > ssh ithaca # (for example) check other HPC platforms where password-less ssh is feasible. - After launching autosubmit, one must be aware of login expiry limit and policy (if applicable for any HPC) and renew the login access accordingly (by using token/key etc) before expiry. HOW TO MONITOR EXPERIMENT ========================= > autosubmit monitor -h > autosubmit monitor cxxx or > autosubmit monitor cxxx -o png Above generated plot with date & time stamp can be found at: /cfu/autosubmit/cxxx/plot/cxxx_date_time.pdf or /cfu/autosubmit/cxxx/plot/cxxx_date_time.png HOW TO RESTART EXPERIMENT ========================= > autosubmit recovery -h > autosubmit recovery cxxx -s # saving the pickle file > nohup autosubmit run cxxx & FULL DOCUMENTATION AND HOW TOs ============================== Check the Autosubmit documentation provided in the docs/ folder of the package, in PDF format. Check the online documentation in the following web page: http://www.bsc.es/projects/earthscience/autosubmit/