README 3.41 KB
Newer Older
Autosubmit is a tool to create, manage and monitor experiments using
jlope2's avatar
jlope2 committed
computing clusters, HPC platforms and supercomputers remotely via ssh.
Domingo Manubens-Gil's avatar
Domingo Manubens-Gil committed
Get involved or contact us:

    Autosubmit GitLab:	        https://earth.bsc.es/gitlab/es/autosubmit
    Autosubmit Mailing List:	autosubmit@bsc.es
Domingo Manubens-Gil's avatar
Domingo Manubens-Gil committed
    
How to cite Autosubmit:

    D. Manubens-Gil, J. Vegas-Regidor, C. Prodhomme, O. Mula-Valls and F. J. Doblas-Reyes,
    "Seamless management of ensemble climate prediction  experiments on HPC platforms," 
    2016 International Conference on High Performance Computing & Simulation (HPCS), 
    Innsbruck, 2016, pp. 895-900. doi: 10.1109/HPCSim.2016.7568429
Muhammad Asif's avatar
 
Muhammad Asif committed

HOW TO DEPLOY/SETUP AUTOSUBMIT FRAMEWORK
========================================
  with the following Operating Systems:
   * Linux Debian
jlope2's avatar
jlope2 committed
   * Linux OpenSUSE
jlope2's avatar
jlope2 committed
  on the following HPC platforms:
   * Stargate (IC3 machine)
   * MareNostrum (BSC machine)
   * MareNostrum3 (BSC machine)
   * HECToR (EPCC machine)
   * Lindgren (PDC machine)
   * C2A (ECMWF machine)
   * CCA (ECMWF machine)
jlope2's avatar
jlope2 committed
- Pre-requisites: These packages (bash, python2, sqlite3, git-scm > 1.8.2, subversion, dialog*) must be available at local
jlope2's avatar
jlope2 committed
  machine. These packages (argparse, dateutil, pyparsing, numpy, pydotplus, matplotlib, paramiko, saga-python,
jlope2's avatar
jlope2 committed
  python2-pythondialog*, mock, portalocker) must be available for python runtime. And the machine is also able to access
jlope2's avatar
jlope2 committed
  HPC platforms via password-less ssh.
jlope2's avatar
jlope2 committed
  *: optional

- Install Autosubmit
   > pip install autosubmit
  or download, unpack and "python setup.py install"

- Create a repository for experiments: Say for example "/cfu/autosubmit" then "autosubmit configure" and follow
  instructions
- Create a blank database: "autosubmit install"
HOW TO USE AUTOSUBMIT
=====================
> autosubmit -h
> autosubmit expid --HPC ithaca --description "experiment is about..."
Muhammad Asif's avatar
 
Muhammad Asif committed

Muhammad Asif's avatar
Muhammad Asif committed
Say for example, "cxxx" is 4 character based expid generated by system automatically.
First character is a letter, the other three alfa-numeric characters allow to identify uniquely the experiment.
Muhammad Asif's avatar
 
Muhammad Asif committed

Muhammad Asif's avatar
Muhammad Asif committed
> vi /cfu/autosubmit/cxxx/conf/expdef_cxxx.conf
Muhammad Asif's avatar
Muhammad Asif committed
> vi /cfu/autosubmit/cxxx/conf/autosubmit_cxxx.conf
Muhammad Asif's avatar
 
Muhammad Asif committed

> vi /cfu/autosubmit/cxxx/conf/platforms_cxxx.conf
Muhammad Asif's avatar
 
Muhammad Asif committed

> vi /cfu/autosubmit/cxxx/conf/jobs_cxxx.conf
> autosubmit create cxxx
jlope2's avatar
jlope2 committed
> nohup autosubmit run cxxx &
Cautions: 
- Before launching autosubmit check the following stuff:
jlope2's avatar
jlope2 committed
> ssh ithaca # (for example) check other HPC platforms where password-less ssh is feasible.
- After launching autosubmit, one must be aware of login expiry limit and policy (if applicable for any HPC)
and renew the login access accordingly (by using token/key etc) before expiry.
HOW TO MONITOR EXPERIMENT
=========================
Muhammad Asif's avatar
 
Muhammad Asif committed

> autosubmit monitor -h
> autosubmit monitor cxxx
> autosubmit monitor cxxx -o png
Above generated plot with date & time stamp can be found at:
Muhammad Asif's avatar
Muhammad Asif committed
/cfu/autosubmit/cxxx/plot/cxxx_date_time.pdf
Muhammad Asif's avatar
Muhammad Asif committed
/cfu/autosubmit/cxxx/plot/cxxx_date_time.png
HOW TO RESTART EXPERIMENT
=========================
> autosubmit recovery -h
> autosubmit recovery  cxxx -s # saving the pickle file
jlope2's avatar
jlope2 committed
> nohup autosubmit run cxxx &
FULL DOCUMENTATION AND HOW TOs
Check the Autosubmit documentation provided in the docs/ folder of the package, in PDF format.
jvegas's avatar
jvegas committed
Check the online documentation in the following web page: http://www.bsc.es/projects/earthscience/autosubmit/