Autosubmit is a tool to create, manage and monitor experiments using Computing Clusters, HPC's and Supercomputers remotely via ssh. HOW TO DEPLOY/SETUP AUTOSUBMIT FRAMEWORK ======================================== - Autosubmit has been tested: with the following Operating Systems: * Linux Debian on the following HPC's/Clusters: * Ithaca (IC3 machine) * Stargate (IC3 machine) * MareNostrum (BSC machine) * MareNostrum3 (BSC machine) * HECToR (EPCC machine) * Lindgren (PDC machine) * C2A (ECMWF machine) * ARCHER (EPCC machine) - Pre-requisties: These packages (bash, python2, sqlite3, git-scm > 1.8.2, subversion) must be available at local machine. These packages (argparse, dateutil, pyparsing, numpy, pydotplus, matplotlib) must be available for python runtime. And the machine is also able to access HPC's/Clusters via password-less ssh. - Install Autosubmit > pip install autosubmit or download, unpack and "python setup.py install" - Create a repository for experiments: Say for example "/cfu/autosubmit" then "autosubmit configure" and follow instructions - Create a blank database: "autosubmit install" HOW TO USE AUTOSUBMIT ===================== > autosubmit -h > autosubmit expid --HPC ithaca --description "experiment is about..." Say for example, "cxxx" is 4 character based expid generated by system automatically. First character "c" represents the platform such as "i" for ithaca, "b" for bsc, "h" for hector, "l" for lindgren, "e" for ecmwf and "m" for marenostrum3 etc. While rest of three characters "xxx" are to represent unique alphanumeric identity for the experiment. > vi /cfu/autosubmit/cxxx/conf/expdef_cxxx.conf > vi /cfu/autosubmit/cxxx/conf/autosubmit_cxxx.conf > vi /cfu/autosubmit/cxxx/conf/queues_cxxx.conf > vi /cfu/autosubmit/cxxx/conf/jobs_cxxx.conf > autosubmit create cxxx > nohup autosubmit run cxxx >& cxxx_01.log & Cautions: - Before launching autosubmit check the following stuff: > ssh ithaca # say for example similarly check other HPC's where password-less ssh is feasible - After launching autosubmit, one must be aware of login expeiry limit and policy (if applicable for any HPC) and renew the login access accordingly (by using token/key etc) before expiry. HOW TO MONITOR EXPERIMENT ========================= > autosubmit monitor -h > autosubmit monitor cxxx or > autosubmit monitor cxxx -o png Above generated plot with date & time stamp can be found at: /cfu/autosubmit/cxxx/plot/cxxx_date_time.pdf or /cfu/autosubmit/cxxx/plot/cxxx_date_time.png HOW TO RESTART EXPERIMENT ========================= > autosubmit recovery -h > autosubmit recovery cxxx -g # getting/fetching completed files > autosubmit recovery cxxx -s # saving the pickle file > nohup autosubmit run cxxx >& cxxx_02.log & HOW TO RERUN/EXTEND EXPERIMENT ============================== > vi /cfu/autosubmit/cxxx/conf/expdef_cxxx.conf # modify RERUN, CHUNKLIST > autosubmit create cxxx > nohup autosubmit cxxx >& cxxx_03.log &