Autosubmit is a tool to create, manage and monitor experiments by using 
different (configured) HPC's/Clusters remotely via ssh.


HOW TO DEPLOY/SETUP AUTOSUBMIT FRAMEWORK
========================================

- Autosubmit has been tested:

  with the following Operating Systems:
   * Linux Debian

  on the following HPC's/Clusters:
   * Ithaca (IC3's Cluster)
   * MareNostrum (Barcelona Supercomputing Center)
   * HECToR (UK Based supercomputer)
   * Lindgren (Swedish machine)

- Pre-requisties: These packages (python2, python-argparse, python-dateutil, 
python-pydot, python-matplotlib, sqlite3) must be available at local machine.
And the machine is also able to access HPC's/Clusters via password-less ssh.

- Create a repository for experiments: Say for expample "/cfu/autosubmit" then
  edit the repository path into src/dir_config.py, src/expid.py, conf/autosubmit.conf

- Create a blank database: Say for example "autosubmit.db" at above created repository
  and thereafter:
   > cd /cfu/autosubmit
   > sqlite3 autosubmit.db
   sqlite3>.read ../../src/autosubmit.sql
   > chmod 777 autosubmit.db


HOW TO USE AUTOSUBMIT
=====================

> cd src

> python expid.py -h

> python expid.py --new ecearth --HPC ithaca --description "experiment is about..."

Say for example, "chex" is 4 character based expid generated by system.
First character "c" represents the platform such as "i" for ithaca, "b" for
bsc, "h" for hector etc. While the rest of three characters are to represent 
unique hexadecimal number for the experiment.

> vi /cfu/autosubmit/chex/conf/expdef_chex.conf

> vi /cfu/autosubmit/chex/conf/autosubmit_chex.conf

> python create_exp.py chex

> ./setupexp.sh -e chex

> nohup python autosubmit.py chex >& chex_01.log &


HOW TO MONITOR EXPERIMENT
=========================

> cd src

> python monitor.py -h

> python monitor.py -e chex -j job_list -o pdf
or
> python monitor.py -e chex -j job_list -o png

Above generated plot with date & time stamp can be found at:

/cfu/autosubmit/chex/plot/chex_date_time.pdf
or 
/cfu/autosubmit/chex/plot/chex_date_time.png


HOW TO RESTART EXPERIMENT
=========================

> cd src

> python recovery.py -h

> python recovery.py -e chex -j job_list -g # getting/fetching completed files

> python recovery.py -e chex -j job_list -s # saving the pickle file

> nohup python autosubmit.py chex >& chex_02.log &


HOW TO RERUN/EXTEND EXPERIMENT
==============================

> cd src

> vi /cfu/autosubmit/chex/conf/expdef_chex.conf # modify RERUN, CHUNKLIST

> python create_exp.py chex

> nohup python autosubmit.py chex >& chex_03.log &

Monitor for RERUN
------------------

> python monitor.py -e chex -j rerun_job_list -o pdf

Recovery for RERUN
-------------------

> python recovery.py -e chex -j rerun_job_list -g 

> python recovery.py -e chex -j rerun_job_list -s