|
|
|
|
|
## Introduction
|
|
## Introduction
|
|
|
|
|
|
[Autosubmit](https://earth.bsc.es/gitlab/es/autosubmit) is a workflow manager developed in-house at BSC. It handles the submission of jobs to specified HPC platforms, and job dependencies. We can also add different ‘chunks’ (by variable, start date, region…) to each job.
|
|
[Autosubmit](https://earth.bsc.es/gitlab/es/autosubmit) is a workflow manager developed in-house at BSC. It handles the submission of jobs to specified HPC platforms, and job dependencies. We can also add different ‘chunks’ (by variable, start date, region…) to each job.
|
|
|
|
|
|
In the Verification Suite, we use Autosubmit to perform the same analysis for independent datasets in a more practical way. Users can now create recipes with multiple systems, start dates, variables and regions to be processed independently and in parallel. You can find an example in the 'Recipes' section in this page.
|
|
In SUNSET, we use Autosubmit to perform the same "[embarrassingly parallel](https://www.wikiwand.com/en/Embarrassingly_parallel)" analyses for independent data in a more practical way. Users can create recipes with multiple systems, start dates, variables and regions, which will be decided into 'atomic recipes'. The analysis of each atomic recipe can then run independently on the HPC platform. You can find an example of how to define your recipe in the 'Recipes' section in this page.
|
|
|
|
|
|
We have implemented Autosubmit configuration file templates that can be used to create and run experiments in a user-friendly way through a few parameters that can be specified in the recipe. Find the details in the 'Instructions' section.
|
|
We have implemented Autosubmit configuration file templates that can be used to create and run experiments in a user-friendly way through a few parameters that can be specified in the recipe. Find the details in the 'Instructions' section.
|
|
|
|
|
... | @@ -28,11 +29,11 @@ Run: |
... | @@ -28,11 +29,11 @@ Run: |
|
Terminal: yes
|
|
Terminal: yes
|
|
filesystem: esarchive
|
|
filesystem: esarchive
|
|
output_dir: /esarchive/scratch/vagudets/auto-s2s-outputs/ # replace with the directory where you want to save the outputs
|
|
output_dir: /esarchive/scratch/vagudets/auto-s2s-outputs/ # replace with the directory where you want to save the outputs
|
|
code_dir: /esarchive/scratch/vagudets/repos/sunset/ # replace with the directory where your code is
|
|
code_dir: /esarchive/scratch/vagudets/repos/auto-s2s/ # replace with the directory where your code is
|
|
autosubmit: yes
|
|
autosubmit: yes
|
|
# fill only if using autosubmit
|
|
# fill only if using autosubmit
|
|
auto_conf:
|
|
auto_conf:
|
|
script: /esarchive/scratch/vagudets/repos/sunset/modules/test_parallel_workflow.R # replace with the path to your script
|
|
script: /esarchive/scratch/vagudets/repos/auto-s2s/modules/test_parallel_workflow.R # replace with the path to your script
|
|
expid: a5no # replace with your EXPID
|
|
expid: a5no # replace with your EXPID
|
|
hpc_user: bsc32762 # replace with your hpc username
|
|
hpc_user: bsc32762 # replace with your hpc username
|
|
wallclock: 01:00 # hh:mm
|
|
wallclock: 01:00 # hh:mm
|
... | @@ -44,31 +45,28 @@ Run: |
... | @@ -44,31 +45,28 @@ Run: |
|
notify_failed: yes # notify me by email when a job fails
|
|
notify_failed: yes # notify me by email when a job fails
|
|
```
|
|
```
|
|
|
|
|
|
In order to be able to find and read the resulting recipes properly, you need to create your script following a template. Find an example of a script that works with split recipes here: [Autosubmit Snippet](https://earth.bsc.es/gitlab/es/sunset/-/snippets/113)
|
|
In order to be able to find and read the resulting recipes properly, you need to create your script following a template. Find an example of a script that works with split atomic recipes here: [Autosubmit Snippet](https://earth.bsc.es/gitlab/es/auto-s2s/-/snippets/113)
|
|
|
|
|
|
**Step 3**: On your workstation or on nord3v2, cd to the code directory and run:
|
|
**Step 3**: On your workstation or on Nord3v2, cd to the code directory and run:
|
|
```shell
|
|
```shell
|
|
source MODULES
|
|
bash launch_SUNSET.sh <path_to_your_recipe> <path_to_your_script>
|
|
Rscript split.R <path_to_your_recipe>
|
|
|
|
```
|
|
```
|
|
This will create an output directory, split your recipe into atomic recipes for independent verifications, and modify the configuration of your experiment according to the parameters in the recipe.
|
|
This will create an output directory, split your recipe into atomic recipes for independent verifications, and modify the configuration of your experiment according to the parameters in the recipe.
|
|
|
|
|
|
> _Tip_: By default, a unique timestamp ID is added to the output directory in your recipe before saving all of the logs and the outputs, in order to avoid overwriting previous files when you run the same recipe again. If you do not want this unique ID added to your output directory, you can add the `--disable_unique_ID` flag:
|
|
> _Tip_: By default, a unique timestamp ID is added to the output directory in your recipe before saving all of the logs and the outputs, in order to avoid overwriting previous files when you run the same recipe again. If you do not want this unique ID added to your output directory, you can add the `--disable_unique_ID` flag:
|
|
>
|
|
>
|
|
> ```shell
|
|
> ```shell
|
|
> Rscript split.R <path_to_your_recipe> --disable_unique_ID
|
|
> bash launch_SUNSET.sh <path_to_your_recipe> <path_to_your_script> --disable_unique_ID
|
|
> ```
|
|
> ```
|
|
|
|
|
|
At the end of the script, some instructions will be written in the terminal explaining what you need to do to finish the set-up and start running your experiment. Follow them.
|
|
At the end of the script, some instructions will be written in the terminal explaining what you need to do to finish the set-up and start running your experiment. Follow them.
|
|
|
|
|
|
**Step 4**: You're done! You can and check the status of your experiment in the [Autosubmit GUI](https://earth.bsc.es/autosubmitapp/). If your experiment doesn't show up at first, try disabling the "Only Active" option next to the search bar.
|
|
**Step 4**: You're done! You can and check the status of your experiment in the [Autosubmit GUI](https://earth.bsc.es/autosubmitapp/). If your experiment doesn't show up in the GUI, try disabling the "Only Active" option next to the search bar.
|
|
|
|
|
|
## Recipes
|
|
## Recipes
|
|
|
|
|
|
Here is a complete example of a 'main' recipe that can be split for independent verifications.
|
|
Here is a complete example of a 'main' recipe that can be split for independent verifications.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```yaml
|
|
```yaml
|
|
################################################################################
|
|
################################################################################
|
|
## RECIPE DESCRIPTION
|
|
## RECIPE DESCRIPTION
|
... | @@ -80,29 +78,29 @@ Description: |
... | @@ -80,29 +78,29 @@ Description: |
|
## ANALYSIS CONFIGURATION
|
|
## ANALYSIS CONFIGURATION
|
|
################################################################################
|
|
################################################################################
|
|
Analysis:
|
|
Analysis:
|
|
Horizon: Seasonal
|
|
Horizon: 'Seasonal'
|
|
Variables: # A list of variables and their frequency
|
|
Variables: # A list of variables and their frequency
|
|
- {name: tas, freq: monthly_mean}
|
|
- {name: 'tas', freq: 'monthly_mean'}
|
|
- {name: prlr, freq: monthly_mean}
|
|
- {name: 'prlr', freq: 'monthly_mean'}
|
|
Datasets:
|
|
Datasets:
|
|
System: # A list of systems
|
|
System: # A list of systems
|
|
- {name: Meteo-France-System7}
|
|
- {name: 'Meteo-France-System7'}
|
|
- {name: ECMWF-SEAS5}
|
|
- {name: 'ECMWF-SEAS5'}
|
|
Multimodel: False # single option
|
|
Multimodel: no # single option
|
|
Reference:
|
|
Reference:
|
|
- {name: ERA5}
|
|
- {name: 'ERA5'}
|
|
Time:
|
|
Time:
|
|
sdate: # A list of start dates
|
|
sdate: # A list of start dates
|
|
- '1101'
|
|
- '1101'
|
|
- '1201'
|
|
- '1201'
|
|
fcst_year: '2020'
|
|
fcst_year: 2020
|
|
hcst_start: '1993'
|
|
hcst_start: 1993
|
|
hcst_end: '2016'
|
|
hcst_end: 2016
|
|
ftime_min: 1
|
|
ftime_min: 1
|
|
ftime_max: 6
|
|
ftime_max: 6
|
|
Region: # A list of regions. If there's more than one region, they need to have a 'name'.
|
|
Region: # A list of regions. If there's more than one region, they need to have a 'name'.
|
|
- {name: "global", latmin: -90, latmax: 90, lonmin: 0, lonmax: 359.9}
|
|
- {name: 'global', latmin: -90, latmax: 90, lonmin: 0, lonmax: 359.9}
|
|
- {name: "nino34", latmin: -5, latmax: 5, lonmin: -10, lonmax: 60}
|
|
- {name: 'nino34', latmin: -5, latmax: 5, lonmin: -10, lonmax: 60}
|
|
Regrid:
|
|
Regrid:
|
|
method: bilinear
|
|
method: bilinear
|
|
type: to_system
|
|
type: to_system
|
... | @@ -115,7 +113,7 @@ Analysis: |
... | @@ -115,7 +113,7 @@ Analysis: |
|
percentiles: [[1/3, 2/3], [1/10, 9/10]]
|
|
percentiles: [[1/3, 2/3], [1/10, 9/10]]
|
|
ncores: 7
|
|
ncores: 7
|
|
remove_NAs: yes
|
|
remove_NAs: yes
|
|
Output_format: S2S4E
|
|
Output_format: 'S2S4E'
|
|
################################################################################
|
|
################################################################################
|
|
## Run CONFIGURATION
|
|
## Run CONFIGURATION
|
|
################################################################################
|
|
################################################################################
|
... | @@ -124,16 +122,17 @@ Run: |
... | @@ -124,16 +122,17 @@ Run: |
|
Terminal: yes
|
|
Terminal: yes
|
|
filesystem: esarchive
|
|
filesystem: esarchive
|
|
output_dir: /esarchive/scratch/vagudets/auto-s2s-outputs/ # replace with the directory where you want to save the outputs
|
|
output_dir: /esarchive/scratch/vagudets/auto-s2s-outputs/ # replace with the directory where you want to save the outputs
|
|
code_dir: /esarchive/scratch/vagudets/repos/sunset/ # replace with the directory where your code is
|
|
code_dir: /esarchive/scratch/vagudets/repos/auto-s2s/ # replace with the directory where your code is
|
|
autosubmit: yes
|
|
autosubmit: yes
|
|
# fill only if using autosubmit
|
|
# fill only if using autosubmit
|
|
auto_conf:
|
|
auto_conf:
|
|
script: /esarchive/scratch/vagudets/repos/sunset/modules/test_parallel_workflow.R # replace with the path to your script
|
|
script: /esarchive/scratch/vagudets/repos/auto-s2s/modules/test_parallel_workflow.R # replace with the path to your script
|
|
expid: a5no # replace with your EXPID
|
|
expid: a5no # replace with your EXPID
|
|
hpc_user: bsc32762 # replace with your hpc username
|
|
hpc_user: bsc32762 # replace with your hpc username
|
|
wallclock: 01:00 # hh:mm
|
|
wallclock: 01:00 # hh:mm
|
|
processors_per_job: 8
|
|
processors_per_job: 8
|
|
platform: nord3v2
|
|
platform: nord3v2
|
|
|
|
custom_directives: ['#SBATCH --exclusive', '#SBATCH --constraint=medmem'] # custom directives for the jobs
|
|
email_notifications: yes # enable/disable email notifications. Change it if you want to.
|
|
email_notifications: yes # enable/disable email notifications. Change it if you want to.
|
|
email_address: victoria.agudetse@bsc.es # replace with your email address
|
|
email_address: victoria.agudetse@bsc.es # replace with your email address
|
|
notify_completed: no # notify me by email when a job finishes
|
|
notify_completed: no # notify me by email when a job finishes
|
... | | ... | |