Skip to content
GitLab

Autosubmit · Changes

Page history
Update Autosubmit authored by vagudets's avatar vagudets
Show whitespace changes
Inline Side-by-side
Autosubmit.md
View page @ a77f4445
## Introduction
[Autosubmit](https://earth.bsc.es/gitlab/es/autosubmit) is a workflow manager developed in-house at BSC. It handles the submission of jobs to specified HPC platforms, and job dependencies. We can also add different ‘chunks’ (by variable, start date, region…) to each job.
In the Verification Suite, we use Autosubmit to perform the same analysis for independent datasets in a more practical way. Users can now create recipes with multiple systems, start dates, variables and regions to be processed independently and in parallel. You can find an example in the 'Recipes' section in this page.
In SUNSET, we use Autosubmit to perform the same "[embarrassingly parallel](https://www.wikiwand.com/en/Embarrassingly_parallel)" analyses for independent data in a more practical way. Users can create recipes with multiple systems, start dates, variables and regions, which will be decided into 'atomic recipes'. The analysis of each atomic recipe can then run independently on the HPC platform. You can find an example of how to define your recipe in the 'Recipes' section in this page.
We have implemented Autosubmit configuration file templates that can be used to create and run experiments in a user-friendly way through a few parameters that can be specified in the recipe. Find the details in the 'Instructions' section.
......@@ -28,11 +29,11 @@ Run:
Terminal: yes
filesystem: esarchive
output_dir: /esarchive/scratch/vagudets/auto-s2s-outputs/ # replace with the directory where you want to save the outputs
code_dir: /esarchive/scratch/vagudets/repos/sunset/ # replace with the directory where your code is
code_dir: /esarchive/scratch/vagudets/repos/auto-s2s/ # replace with the directory where your code is
autosubmit: yes
# fill only if using autosubmit
auto_conf:
script: /esarchive/scratch/vagudets/repos/sunset/modules/test_parallel_workflow.R # replace with the path to your script
script: /esarchive/scratch/vagudets/repos/auto-s2s/modules/test_parallel_workflow.R # replace with the path to your script
expid: a5no # replace with your EXPID
hpc_user: bsc32762 # replace with your hpc username
wallclock: 01:00 # hh:mm
......@@ -44,31 +45,28 @@ Run:
notify_failed: yes # notify me by email when a job fails
```
In order to be able to find and read the resulting recipes properly, you need to create your script following a template. Find an example of a script that works with split recipes here: [Autosubmit Snippet](https://earth.bsc.es/gitlab/es/sunset/-/snippets/113)
In order to be able to find and read the resulting recipes properly, you need to create your script following a template. Find an example of a script that works with split atomic recipes here: [Autosubmit Snippet](https://earth.bsc.es/gitlab/es/auto-s2s/-/snippets/113)
**Step 3**: On your workstation or on nord3v2, cd to the code directory and run:
**Step 3**: On your workstation or on Nord3v2, cd to the code directory and run:
```shell
source MODULES
Rscript split.R <path_to_your_recipe>
bash launch_SUNSET.sh <path_to_your_recipe> <path_to_your_script>
```
This will create an output directory, split your recipe into atomic recipes for independent verifications, and modify the configuration of your experiment according to the parameters in the recipe.
> _Tip_: By default, a unique timestamp ID is added to the output directory in your recipe before saving all of the logs and the outputs, in order to avoid overwriting previous files when you run the same recipe again. If you do not want this unique ID added to your output directory, you can add the `--disable_unique_ID` flag:
>
> ```shell
> Rscript split.R <path_to_your_recipe> --disable_unique_ID
> bash launch_SUNSET.sh <path_to_your_recipe> <path_to_your_script> --disable_unique_ID
> ```
At the end of the script, some instructions will be written in the terminal explaining what you need to do to finish the set-up and start running your experiment. Follow them.
**Step 4**: You're done! You can and check the status of your experiment in the [Autosubmit GUI](https://earth.bsc.es/autosubmitapp/). If your experiment doesn't show up at first, try disabling the "Only Active" option next to the search bar.
**Step 4**: You're done! You can and check the status of your experiment in the [Autosubmit GUI](https://earth.bsc.es/autosubmitapp/). If your experiment doesn't show up in the GUI, try disabling the "Only Active" option next to the search bar.
## Recipes
Here is a complete example of a 'main' recipe that can be split for independent verifications.
```yaml
################################################################################
## RECIPE DESCRIPTION
......@@ -80,29 +78,29 @@ Description:
## ANALYSIS CONFIGURATION
################################################################################
Analysis:
Horizon: Seasonal
Horizon: 'Seasonal'
Variables: # A list of variables and their frequency
- {name: tas, freq: monthly_mean}
- {name: prlr, freq: monthly_mean}
- {name: 'tas', freq: 'monthly_mean'}
- {name: 'prlr', freq: 'monthly_mean'}
Datasets:
System: # A list of systems
- {name: Meteo-France-System7}
- {name: ECMWF-SEAS5}
Multimodel: False # single option
- {name: 'Meteo-France-System7'}
- {name: 'ECMWF-SEAS5'}
Multimodel: no # single option
Reference:
- {name: ERA5}
- {name: 'ERA5'}
Time:
sdate: # A list of start dates
- '1101'
- '1201'
fcst_year: '2020'
hcst_start: '1993'
hcst_end: '2016'
fcst_year: 2020
hcst_start: 1993
hcst_end: 2016
ftime_min: 1
ftime_max: 6
Region: # A list of regions. If there's more than one region, they need to have a 'name'.
- {name: "global", latmin: -90, latmax: 90, lonmin: 0, lonmax: 359.9}
- {name: "nino34", latmin: -5, latmax: 5, lonmin: -10, lonmax: 60}
- {name: 'global', latmin: -90, latmax: 90, lonmin: 0, lonmax: 359.9}
- {name: 'nino34', latmin: -5, latmax: 5, lonmin: -10, lonmax: 60}
Regrid:
method: bilinear
type: to_system
......@@ -115,7 +113,7 @@ Analysis:
percentiles: [[1/3, 2/3], [1/10, 9/10]]
ncores: 7
remove_NAs: yes
Output_format: S2S4E
Output_format: 'S2S4E'
################################################################################
## Run CONFIGURATION
################################################################################
......@@ -124,16 +122,17 @@ Run:
Terminal: yes
filesystem: esarchive
output_dir: /esarchive/scratch/vagudets/auto-s2s-outputs/ # replace with the directory where you want to save the outputs
code_dir: /esarchive/scratch/vagudets/repos/sunset/ # replace with the directory where your code is
code_dir: /esarchive/scratch/vagudets/repos/auto-s2s/ # replace with the directory where your code is
autosubmit: yes
# fill only if using autosubmit
auto_conf:
script: /esarchive/scratch/vagudets/repos/sunset/modules/test_parallel_workflow.R # replace with the path to your script
script: /esarchive/scratch/vagudets/repos/auto-s2s/modules/test_parallel_workflow.R # replace with the path to your script
expid: a5no # replace with your EXPID
hpc_user: bsc32762 # replace with your hpc username
wallclock: 01:00 # hh:mm
processors_per_job: 8
platform: nord3v2
custom_directives: ['#SBATCH --exclusive', '#SBATCH --constraint=medmem'] # custom directives for the jobs
email_notifications: yes # enable/disable email notifications. Change it if you want to.
email_address: victoria.agudetse@bsc.es # replace with your email address
notify_completed: no # notify me by email when a job finishes
......