Skip to content
GitLab

Home · Changes

Page history
Update home authored by Miguel Castrillo's avatar Miguel Castrillo
Hide whitespace changes
Inline Side-by-side
home.md
View page @ 305a45ec
......@@ -2,21 +2,17 @@ Autosubmit launches and monitors experiments on any platform used at Earth Scien
## Description
Autosubmit is a Python-based workflow manager to create, manage and monitor complex tasks involving different sub steps, such as scientific computational experiments. These workflows may involve multiple computing systems for its completion, from HPCs to post-processing clusters or workstations. Autosubmit is capable of orchestrating all the tasks integrating the workflow, by managing their dependencies, interfacing with all the platforms involved and handling eventual errors.
Autosubmit is a Python-based workflow manager to create, manage and monitor complex tasks involving different sub steps, such as scientific computational experiments. These workflows may involve multiple computing systems for its completion, from HPCs to post-processing clusters or workstations. Autosubmit is capable of orchestrating all the tasks integrating the workflow, by managing their dependencies, interfacing with all the platforms involved and handling eventual errors.
Autosubmit is currently used at Barcelona Supercomputing Centre (BSC) to run models (EC-Earth, MONARCH, NEMO, CALIOPE, HERMES...), operational toolchains (S2S4E), data-download workflows (ECMWF MARS), and many other.
Autosubmit has been used to manage models running at supercomputers in BSC, ECMWF, IC3, CESGA, EPCC, PDC and OLCF.
Autosubmit is the only existing tool that satisfies the following requirements from the weather and climate community:
* **Automatisation**: Job submission to machines and dependencies between jobs are managed by Autosubmit. No user intervention is needed.
* **Data provenance**: Assigns unique identifiers for each experiment and stores information about model version, experiment configuration and computing facilities used in the whole process.
* **Failure tolerance**: Automatic retrials and ability to rerun chunks in case of corrupted or missing data.
* **Resource management**: Autosubmit manages supercomputer particularities, allowing users to run their experiments without having to adapt their codes for that purpose. Autosubmit also allows to submit tasks from the same experiment to different platforms.
And some of its highlights are:
......@@ -31,22 +27,27 @@ And some of its highlights are:
## How to cite
* D. Manubens-Gil, J. Vegas-Regidor, C. Prodhomme, O. Mula-Valls and F. J. Doblas-Reyes, “Seamless management of ensemble climate prediction experiments on HPC platforms,” 2016 International Conference on High Performance Computing & Simulation (HPCS), Innsbruck, 2016, pp. 895-900. doi: 10.1109/HPCSim.2016.7568429 ([PDF](https://earth.bsc.es/wiki/lib/exe/fetch.php?media=publications:dmanubens_hpcs_2016.pdf))
* D. Manubens-Gil, J. Vegas-Regidor, C. Prodhomme, O. Mula-Valls and F. J. Doblas-Reyes, “Seamless management of ensemble climate prediction experiments on HPC platforms,” 2016 International Conference on High Performance Computing & Simulation (HPCS), Innsbruck, 2016, pp. 895-900. doi: 10.1109/HPCSim.2016.7568429 ([PDF](https://earth.bsc.es/wiki/lib/exe/fetch.php?media=publications:dmanubens_hpcs_2016.pdf))
## Contact
## Contact persons
Code developed at [Barcelona Supercomputing Center](https://www.bsc.es) (BSC-CNS).
Code developed at [Barcelona Supercomputing Center](https://www.bsc.es) (BSC-CNS).
External support mail: [support-autosubmit@bsc.es](mailto:support-autosubmit@bsc.es)
Developers:
* Daniel Beltrán Mora - daniel.beltran@bsc.es
* Wilmer Uruchi Ticona - wilmer.uruchi@bsc.es
Developers:
Maintainer:
* Daniel Beltrán Mora - daniel.beltran@bsc.es
* Daniel Beltrán Mora - [daniel.beltran@bsc.es](mailto:daniel.beltran@bsc.es)
* Wilmer Uruchi Ticona - [wilmer.uruchi@bsc.es](mailto:wilmer.uruchi@bsc.es)
Maintainer:
* Daniel Beltrán Mora - [daniel.beltran@bsc.es](mailto:daniel.beltran@bsc.es)
Former Developers:
* Domingo Manubens Gil
* Javier Vegas-Regidor - javier.vegas@bsc.es
* Javier Vegas-Regidor - [javier.vegas@bsc.es](mailto:javier.vegas@bsc.es)
* Larissa Batista Leite
* Joan López
* Oriol Mula-Mula Valls
......@@ -54,14 +55,15 @@ Former Developers:
### Requirements
Autosubmit has been tested with the following Operating Systems:
* Linux Debian
* Linux OpenSUSE
**Pre-requisites**:
* Linux Debian
* Linux OpenSUSE
**Pre-requisites**:
- These packages (bash, python2, sqlite3, git-scm > 1.8.2, subversion, dialog\*) must be available at local machine.
- These packages (argparse, dateutil, pyparsing, numpy, pydotplus, matplotlib, paramiko, python2-pythondialog\*, mock, portalocker) must be available for python runtime. The machine needs to be able to access HPC platforms via password-less ssh.
- These packages (bash, python2, sqlite3, git-scm > 1.8.2, subversion, dialog*) must be available at local machine.
- These packages (argparse, dateutil, pyparsing, numpy, pydotplus, matplotlib, paramiko, python2-pythondialog*, mock, portalocker) must be available for python runtime. The machine needs to be able to access HPC platforms via password-less ssh.
## Usage
#### [User guide](http://autosubmit.readthedocs.io/en/latest/introduction.html)
......@@ -82,10 +84,10 @@ Autosubmit has been tested with the following Operating Systems:
#### [Deploy new version](Deployment)
#### [Branching conventions (Earth wiki)](https://earth.bsc.es/wiki/doku.php?id=library:git)
#### [Branching conventions (Earth wiki)](https://earth.bsc.es/wiki/doku.php?id=library:git)
#### [Style guide](https://earth.bsc.es/wiki/doku.php?id=tools:style_guides:python)
#### [Style guide](https://earth.bsc.es/wiki/doku.php?id=tools:style_guides:python)
#### [Testing Suite](Testing_Suite)
#### [Testing Suite](Testing_Suite)
#### [Possible Operational Problems and Solutions](Possible-Operational-Problems-and-Solutions)
#### [Possible Operational Problems and Solutions](Possible-Operational-Problems-and-Solutions)
\ No newline at end of file