User Tools

Site Tools


tools:autosubmit

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
tools:autosubmit [2020/07/02 15:36]
mcastril [Repository]
tools:autosubmit [2023/08/24 16:25] (current)
pgoitia
Line 1: Line 1:
- 
- 
- 
-__This wiki is being updated (June 2020). Most of their content will be moved to GitLab wiki. You can check the Useful links section below.  
-__ 
- 
 ==== Objective ==== ==== Objective ====
  
Line 12: Line 6:
 ==== Repository ==== ==== Repository ====
  
-The repository and issue tracker of Autosubmit is here+The repository and issue tracker of Autosubmit (and the API and the GUI) is here
  
- [[https://earth.bsc.es/gitlab/es/autosubmit.git]]+  * [[https://earth.bsc.es/gitlab/es/autosubmit.git|Autosubmit GitLab]] 
 +  * [[https://earth.bsc.es/gitlab/es/autosubmit_api|Autosubmit API GitLab]] 
 +  * [[https://earth.bsc.es/gitlab/es/autosubmitreact|Autosubmit GUI GitLab]] 
 + 
 +=== Using Autosubmit at the Earth Sciences department === 
 + 
 +  * [[https://earth.bsc.es/gitlab/es/autosubmit/-/wikis/Running-Autosubmit-in-Earth-Sciences|Instructions in Autosubmit wiki]]
  
 === Useful links === === Useful links ===
 +  * [[https://earth.bsc.es/gitlab/es/autosubmit/-/wikis/home|Autosubmit GitLab wiki]]
 +  * [[https://autosubmit.readthedocs.io/en/master/introduction/index.html|Autosubmit Documentation]] 
 +  * [[https://earth.bsc.es/wiki/doku.php?id=tools:autosubmit:tutorials|Autosubmit Tutorials]]
 +  * [[https://autosubmitgui.bsc.es/presentation|Autosubmit App (GUI)]] (public demo)
 +  * [[http://bscesweb04.bsc.es/autosubmitapp/|Autosubmit App (GUI)]] (only accessible from inside BSC network)
 +  * [[http://bscesautosubmit01.bsc.es:8000/autosubmit_v3|List of experiments v3]] (LEGACY, only accessible from inside BSC network)
  
-  * [[https://autosubmit.readthedocs.io/en/latest/introduction.html|Autosubmit Documentation]]  
-  * [[http://bscesweb04.bsc.es/autosubmitapp/|Autosubmit App (GUI)]] (only accessible form inside BSC network) 
-  *  [[http://bscesautosubmit01.bsc.es:8000/autosubmit_v3|List of experiments v3]] (only accessible form inside BSC network) 
  
  
Line 26: Line 29:
  
 [[http://ieeexplore.ieee.org/document/7568429/|IEEE]] {{:publications:dmanubens_hpcs_2016.pdf|pdf}} D. Manubens-Gil, J. Vegas-Regidor, C. Prodhomme, O. Mula-Valls and F. J.  Doblas-Reyes, "Seamless management of ensemble climate prediction  experiments on HPC platforms," 2016 International Conference on High Performance Computing & Simulation (HPCS), Innsbruck, 2016, pp. 895-900. [[http://ieeexplore.ieee.org/document/7568429/|IEEE]] {{:publications:dmanubens_hpcs_2016.pdf|pdf}} D. Manubens-Gil, J. Vegas-Regidor, C. Prodhomme, O. Mula-Valls and F. J.  Doblas-Reyes, "Seamless management of ensemble climate prediction  experiments on HPC platforms," 2016 International Conference on High Performance Computing & Simulation (HPCS), Innsbruck, 2016, pp. 895-900.
-doi: 10.1109/HPCSim.2016.7568429 +https://doi.org/10.1109/HPCSim.2016.7568429
- +
-==== Description ==== +
- +
- +
- +
-=== General Description === +
- +
- +
-== Introduction == +
- +
-A typical climate forecast experiment is a run of a climate model over a supercomputer having variable range of forecast length from a few months to a few years. And an experiment may have one or more than one start-dates and every start-date may comprise of single or many members. The full length of forecasting period for the experiment could be divided into number of chunks of fixed forecast length by exploiting the available options of model restart. Furthermore, in the context of computing operations, every chunk could have two big sections; parallel section where the actually model run would be performed by using computing cores of supercomputer and serial section(s) for performing other necessary operations like post-processing of the model output, archiving the model output and cleaning the disk space for the smooth proceeding of the experiment.  +
-{{file:experiment_new.png}} +
- +
-As we could see in the sample experiment which consists of 10 start-dates from 1960 to 2005 where every start-date is independent of each other and starting after every 5 years while each start-date comprise of 5 members. Every member is also independent and has been divided into 10 chunks which are dependent on each other. Now let us suppose that the forecast length for each chunk is one year and every chunk comprises of three types of jobs; a simulation (Sim), a post-processing (Post) and an archiving and cleaning job (Clean). Therefore with this typical exemplary experiment, one start-date with one member comprise of 30 jobs and eventually 1500 jobs will be run in total for the completion of the experiment. In short, there is a need of a system to automate such type of typical experiments and optimize the use of resources. +
- +
- +
-== Goal == +
- +
-Autosubmit is a tool to manage and monitor climate forecasting experiments by using supercomputers remotely and achieve the following goals: +
- +
-  *  Efficient handling of highly dependent jobs +
-  *  Optimum utilization of available computing resources +
-  *  Ease of starting, stopping and live monitoring of experiments +
-  *  Auto restarting the experiment or some part of experiment in case of failure +
-  *  Use of database for experiment creation and assigning automatic experiment identity +
-  *  Ability to reproduce the completed experiments fully or partially. +
-{{file:autosubmit24.png}} +
- +
- +
-=== Technical Description === +
- +
- +
-== IS-ENES 2 == +
- +
- +
-**A CNRM-CM6 monitoring using Autosubmit** +
- +
- +
-A few members of seasonal forecast experiment using CNRM-CM6 on ECMWF IBM Power 7 has been performed using Autosubmit monitoring. A few day long collaboration at IC3 has been sufficient to adapt the existing CNRM workflow script to Autosubmit non-intrusive requirements. Nevertheless, a more comprehensive work would be necessary to fully exploit Autosubmit capabilities to monitor and control the full workflow (from compiling) on any kind of supercomputer platform. +
- +
-The technical report descirbing the work is available here: http://www.cerfacs.fr/globc/publication/technicalreport/2014/autosubmit_cnrm-cm.pdf +
- +
- +
-==== Requirements ==== +
- +
-=== How to deploy/setup Autosubmit (v3) === +
- +
-If you need Autosubmit 3.x on a non ES machine, you can download and install it by typing +
- +
-<code>pip install autosubmit</code> +
- +
-And follow installation documentation here: +
- +
-[[http://www.bsc.es/projects/earthscience/autosubmit/installation.html]] +
- +
-=== How to deploy/setup Autosubmit (v2) === +
- +
-If you need an older version (2.x) you must download it by typing: +
- +
-<code>git clone https://earth.bsc.es/gitlab/es/autosubmit.git</code> +
- +
-and then you can switch to the required tag with +
- +
-<code>git checkout tags/<tag_name></code> +
- +
-Pre-requisties: These packages (python2, python-argparse, python-dateutil, python-pydot, python-matplotlib, sqlite3) must be available at local machine. And the machine is also able to access HPC's/Clusters via password-less ssh. +
- +
-Create a repository for experiments: Say for example "/cfu/autosubmit" then edit the repository path into src/dir_config.py, src/expid.py, conf/autosubmit.conf +
-Create a blank database: Say for example "autosubmit.db" at above created repository and thereafter: +
-  > cd /cfu/autosubmit +
-  > sqlite3 autosubmit.db +
-  sqlite3>.read ../../src/autosubmit.sql +
-  > chmod 777 autosubmit.db +
- +
- +
-==== Use ==== +
- +
-User guide: [[http://www.bsc.es/projects/earthscience/autosubmit/]] +
- +
-  *  Autosubmit 3.0+ [[https://readthedocs.org/projects/autosubmit|documentation]] +
- +
-Run Autosubmit in BSC-ES: [[tools:autosubmit:bsces|Here]] +
-  +
- +
-==== Tutorials ==== +
- +
-[[tools:autosubmit:tutorials|Tutorials]]  +
- +
-[[tools:autosubmit:past_tutorials|Past Tutorials]]  +
- +
- +
-==== Presentations ==== +
-  *  Call for Autosubmit users - May 2016 {{:tools:call_for_autosubmit_users_26_05_.pdf|}} --- //[[domingo.manubens@bsc.es|DOMINGO MANUBENS]] 2016/05/26 12:27// +
-  * Autosubmit 3.0.0 CFU presentation {{file:as300.pdf}} --Dmanubens (talk) 18:28, 3 December 2014 (CET) +
-  * Autosubmit 3.0.0 training {{file:as3_training.pdf}} --Dmanubens (talk) 18:32, 3 December 2014 (CET) +
-  * Autosubmit 2.4.1 CFU presentation {{file:as241.pdf}} --Dmanubens (talk) 17:27, 4 July 2014 (CEST) +
-  * Autosubmit and GIT: new projects {{file:asand_git.pdf}} +
-  * Autosubmit 2.3 and GIT {{file:as23and_git.pdf}}  +
-  * Assesment report on Autosubmit, Cylc and ecFlow - {{:tools:is-enes2_d93_v1.0_mp.pdf|}} +
  
 +==== How to cite the Autosubmit Graphical Interface ====
  
 +[[https://joss.theoj.org/papers/10.21105/joss.03049#|JOSS]] Uruchi et al., (2021). Autosubmit GUI: A Javascript-based Graphical User Interface to Monitor Experiments Workflow Execution. Journal of Open Source Software, 6(59), 3049, https://doi.org/10.21105/joss.03049
 ==== Contact ==== ==== Contact ====
  
Line 134: Line 39:
 The coordinator of this project is Miguel Castrillo <miguel.castrillo@bsc.es> The coordinator of this project is Miguel Castrillo <miguel.castrillo@bsc.es>
  
-Miguel Castrillo <miguel.castrillo@bsc.es>, Daniel Beltrán <daniel.beltran@bsc.es>, Wilmer Uruchi <wilmer.uruchi@bsc.es> +Daniel Beltrán <daniel.beltran@bsc.es>, Miguel Castrillo <miguel.castrillo@bsc.es> 
- +
- +
- +
-==== Development ==== +
- +
- +
- +
-=== SCRUM Framework === +
- +
-  * [[https://trello.com/b/0la8i6Fg/autosubmit]] +
- +
-Old version +
- +
-  *  [[tools:scrum|SCRUM Framework]] +
- +
- +
-=== GIT branching scheme === +
- +
-See the following page to check the current branching scheme used within the GIT project 'autosubmit': [[library:git#git_branching_scheme|Git branching scheme]] +
- +
- +
-==== Style Guide ==== +
- +
-You can check the style guide for Autosubmit [[tools:style_guides:python|here]]+
  
-==== Update to the latest autosubmit development version using virtual environment ==== 
  
-Adapted from the wiki page about virtual environments  
- [[ https://earth.bsc.es/wiki/doku.php?id=library:python_venvs ]]  
  
-   > ssh -X bscesautosubmit01 
-   > module purge 
-   > mkdir -p ~/venvs/as_dev 
-   > virtualenv ~/venvs/as_dev 
-   > source ~/venvs/as_dev/bin/activate 
-   > pip install https://earth.bsc.es/gitlab/es/autosubmit/repository/archive.zip?ref=develop 
-   > autosubmit -v  
-    
-In the meantime, if there has been autosubmit development, to update it to the latest version:  
  
-   > ssh -X bscesautosubmit01 
-   > module purge 
-   > source ~/venvs/as_dev/bin/activate 
-   > pip install --upgrade https://earth.bsc.es/gitlab/es/autosubmit/repository/archive.zip?ref=develop 
-   > autosubmit -v     
  
tools/autosubmit.1593704161.txt.gz · Last modified: 2020/07/02 15:36 by mcastril