Autosubmit launches and monitors experiments on any platform used at Earth Sciences Department. A general description of the goal of Autosubmit, how it works, how to install on your computer, user's manual and documentation is available here.
Autosubmit is a Python-based workflow manager to create, manage and monitor complex tasks involving different sub steps, as scientific computational experiments, which may be executed in one or different computing systems, from HPCs to small clusters or workstatoins. Autosubmit is able to orchestrate the tasks composing the workflow while respecting their dependencies and handling errors. It allows to runs different types of experiments, with one or several startdates and ensemble members, allowing to pack different tasks so to better use the resources.
Autosubmit is currently used at Barcelona Supercomputing Centre (BSC) to run models (EC-Earth, MONARCH, NEMO, CALIOPE, HERMES...), operational toolchains (S2S4E), data-download workflows (ECMWF MARS), and many other.
Autosubmit has been used to manage models running at supercomputers in BSC, ECMWF, IC3, CESGA, EPCC, PDC and OLCF.
Autosubmit is the only existing tool that satisfies the following requirements from the weather and climate community:
Automatisation: Job submission to machines and dependencies between jobs are managed by Autosubmit. No user intervention is needed.
Data provenance: Assigns unique identifiers for each experiment and stores information about model version, experiment configuration and computing facilities used in the whole process.
Failure tolerance: Automatic retrials and ability to rerun chunks in case of corrupted or missing data.
Resource management: Autosubmit manages supercomputer particularities, allowing users to run their experiments without having to adapt their codes for that purpose. Autosubmit also allows to submit tasks from the same experiment to different platforms.
And some of its highlights are:
- Efficient handling of highly dependent tasks.
- Optimum utilization of the computing resources.
- Ease of experiment starting, stopping and live monitoring.
- Capability to resume an experiment or some part of it in case of failure.
- Ability to copy experiments and reproduce them fully or partially.
How to cite
D. Manubens-Gil, J. Vegas-Regidor, C. Prodhomme, O. Mula-Valls and F. J. Doblas-Reyes, “Seamless management of ensemble climate prediction experiments on HPC platforms,” 2016 International Conference on High Performance Computing & Simulation (HPCS), Innsbruck, 2016, pp. 895-900. doi: 10.1109/HPCSim.2016.7568429 (PDF)
[BibTeX citation] (Bibtex)
Code developed at Barcelona Supercomputing Center (BSC-CNS).
- Daniel Beltrán Mora - email@example.com
- Domingo Manubens Gil
- Javier Vegas-Regidor - firstname.lastname@example.org
- Larissa Batista Leite
- Joan López
- Oriol Mula-Mula Valls
Autosubmit has been tested with the following Operating Systems:
- Linux Debian
- Linux OpenSUSE
- These packages (bash, python2, sqlite3, git-scm > 1.8.2, subversion, dialog*) must be available at local machine.
- These packages (argparse, dateutil, pyparsing, numpy, pydotplus, matplotlib, paramiko, python2-pythondialog*, mock, portalocker) must be available for python runtime. The machine needs to be able to access HPC platforms via password-less ssh.