|
Autosubmit launches and monitors experiments on any platform used at Earth Sciences Department. A general description of the goal of Autosubmit, how it works, how to install on your computer, user's manual and documentation is available here.
|
|
Autosubmit launches and monitors experiments on any platform used at Earth Sciences Department. A general description of the goal of Autosubmit, how it works, how to install it on your computer, the user's manual, and documentation is available here.
|
|
|
|
|
|
## Description
|
|
## Description
|
|
|
|
|
|
Autosubmit is a Python-based workflow manager to create, manage and monitor complex tasks involving different sub steps, such as scientific computational experiments. These workflows may involve multiple computing systems for its completion, from HPCs to post-processing clusters or workstations. Autosubmit is capable of orchestrating all the tasks integrating the workflow, by managing their dependencies, interfacing with all the platforms involved and handling eventual errors.
|
|
Autosubmit is a Python-based workflow manager to create, manage and monitor complex tasks involving different substeps, such as scientific computational experiments. These workflows may involve multiple computing systems for their completion, from HPCs to post-processing clusters or workstations. Autosubmit can orchestrate all the tasks integrating the workflow by managing their dependencies, interfacing with all the platforms involved, and handling eventual errors.
|
|
|
|
|
|
Autosubmit is currently used at Barcelona Supercomputing Centre (BSC) to run models (EC-Earth, MONARCH, NEMO, CALIOPE, HERMES...), operational toolchains (S2S4E), data-download workflows (ECMWF MARS), and many other.
|
|
Autosubmit is currently used at Barcelona Supercomputing Centre (BSC) to run models (EC-Earth, MONARCH, NEMO, CALIOPE, HERMES...), operational toolchains (S2S4E), data-download workflows (ECMWF MARS), and many other.
|
|
|
|
|
|
Autosubmit has been used to manage models running at supercomputers in BSC, ECMWF, IC3, CESGA, EPCC, PDC and OLCF.
|
|
Autosubmit has been used to manage models running at supercomputers in BSC, ECMWF, IC3, CESGA, EPCC, PDC, and OLCF.
|
|
|
|
|
|
Autosubmit is the only existing tool that satisfies the following requirements from the weather and climate community:
|
|
Autosubmit is the only existing tool that satisfies the following requirements from the weather and climate community:
|
|
|
|
|
|
* **Automatisation**: Job submission to machines and dependencies between jobs are managed by Autosubmit. No user intervention is needed.
|
|
* **Automatisation**: Job submission to machines and dependencies between jobs are managed by Autosubmit. No user intervention is needed.
|
|
* **Data provenance**: Assigns unique identifiers for each experiment and stores information about model version, experiment configuration and computing facilities used in the whole process.
|
|
* **Data provenance**: Assign unique identifiers for each experiment and store information about model version, experiment configuration, and computing facilities used in the process.
|
|
* **Failure tolerance**: Automatic retrials and ability to rerun chunks in case of corrupted or missing data.
|
|
* **Failure tolerance**: Automatic retrials and the ability to rerun chunks in case of corrupted or missing data.
|
|
* **Resource management**: Autosubmit manages supercomputer particularities, allowing users to run their experiments without having to adapt their codes for that purpose. Autosubmit also allows to submit tasks from the same experiment to different platforms.
|
|
* **Resource management**: Autosubmit manages supercomputer particularities, allowing users to run their experiments without having to adapt their codes for that purpose. Autosubmit also allows submitting tasks from the same experiment to different platforms.
|
|
|
|
|
|
And some of its highlights are:
|
|
And some of its highlights are:
|
|
|
|
|
|
* Efficient handling of highly dependent tasks.
|
|
* Efficient handling of highly dependent tasks.
|
|
* Optimum utilization of the computing resources.
|
|
* Optimum utilization of the computing resources.
|
|
* Ease of experiment starting, stopping and live monitoring.
|
|
* Ease of experiment starting, stopping, and live monitoring.
|
|
* Capability to resume an experiment or some part of it in case of failure.
|
|
* Capability to resume an experiment or some part of it in case of failure.
|
|
* Ability to copy experiments and reproduce them fully or partially.
|
|
* Ability to copy experiments and reproduce them fully or partially.
|
|
|
|
|
... | @@ -50,7 +50,7 @@ Former Developers: |
... | @@ -50,7 +50,7 @@ Former Developers: |
|
* Larissa Batista Leite
|
|
* Larissa Batista Leite
|
|
* Joan López
|
|
* Joan López
|
|
* Oriol Mula-Mula Valls
|
|
* Oriol Mula-Mula Valls
|
|
* Wilmer Uruchi Ticona -
|
|
* Wilmer Uruchi Ticona
|
|
|
|
|
|
### Requirements
|
|
### Requirements
|
|
|
|
|
... | @@ -78,8 +78,6 @@ Autosubmit has been tested with the following Operating Systems: |
... | @@ -78,8 +78,6 @@ Autosubmit has been tested with the following Operating Systems: |
|
|
|
|
|
#### [Possible Operational Problems and Solutions](Possible-Operational-Problems-and-Solutions)
|
|
#### [Possible Operational Problems and Solutions](Possible-Operational-Problems-and-Solutions)
|
|
|
|
|
|
#### [Master Change log](https://earth.bsc.es/gitlab/es/autosubmit/-/blob/master/CHANGELOG)
|
|
|
|
|
|
|
|
## Development
|
|
## Development
|
|
|
|
|
|
#### [Development](Development)
|
|
#### [Development](Development)
|
... | | ... | |