|
|
|
## Introduction
|
|
|
|
|
|
|
|
[Autosubmit](https://earth.bsc.es/gitlab/es/autosubmit) is a workflow manager developed in-house at BSC. It handles the submission of jobs to specified HPC platforms, and job dependencies. We can also add different ‘chunks’ (by variable, start date, region…) to each job.
|
|
|
|
|
|
|
|
In the Verification Suite, we use Autosubmit to perform the same analysis for independent datasets in a more practical way. Users can now create recipes with multiple systems, start dates, variables and regions to be processed independently and in parallel. You can find an example in the 'Recipes' section in this page.
|
|
|
|
|
|
|
|
We have implemented Autosubmit configuration file templates that can be used to create and run experiments in a user-friendly way through a few parameters that can be specified in the recipe. Find the details in the 'Instructions' section.
|
|
|
|
|
|
|
|
|
|
|
|
## Instructions
|
|
|
|
|
|
|
|
**Step 1**: Create a new autosubmit experiment. From the terminal, ssh into the autosubmit machine (bscesautosubmit01) and enter the following commands:
|
|
|
|
|
|
|
|
```shell
|
|
|
|
module load autosubmit/3.14.0-foss-2015a-Python-2.7.9
|
|
|
|
autosubmit expid -H nord3v2 -d <Description> # Write anything you like in the description, like "SEAS5 Verification experiment" for example.
|
|
|
|
```
|
|
|
|
|
|
|
|
This will create a new autosubmit experiment which you can use for your verification workflows, with a new 4-letter ID ("EXPID"). Store your EXPID somewhere so you remember it. You will not have to create a new experiment every time, as you can use this EXPID for other verification runs. It's up to you how many you want to create.
|
|
|
|
|
|
|
|
**Step 2**: Create your recipe and script. To run them with autosubmit, you will have to add some configuration parameters to the run section of the recipe, described below:
|
|
|
|
|
|
|
|
NOTE: If you plan to use autosubmit, it is highly recommended that the `output_dir` is NOT inside of the `code_dir`.
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
Run:
|
|
|
|
Loglevel: INFO
|
|
|
|
Terminal: yes
|
|
|
|
filesystem: esarchive
|
|
|
|
output_dir: /esarchive/scratch/vagudets/auto-s2s-outputs/ # replace with the directory where you want to save the outputs
|
|
|
|
code_dir: /esarchive/scratch/vagudets/repos/auto-s2s/ # replace with the directory where your code is
|
|
|
|
autosubmit: yes
|
|
|
|
# fill only if using autosubmit
|
|
|
|
auto_conf:
|
|
|
|
script: /esarchive/scratch/vagudets/repos/auto-s2s/modules/test_parallel_workflow.R # replace with the path to your script
|
|
|
|
expid: a5no # replace with your EXPID
|
|
|
|
hpc_user: bsc32762 # replace with your hpc username
|
|
|
|
wallclock: 01:00 # hh:mm
|
|
|
|
processors_per_job: 8
|
|
|
|
platform: nord3v2
|
|
|
|
email_notifications: yes # enable/disable email notifications. Change it if you want to.
|
|
|
|
email_address: victoria.agudetse@bsc.es # replace with your email address
|
|
|
|
notify_completed: no # notify me by email when a job finishes
|
|
|
|
notify_failed: yes # notify me by email when a job fails
|
|
|
|
```
|
|
|
|
|
|
|
|
In order to be able to find and read the resulting recipes properly, you need to create your script following a template. Find an example of a script that works with split recipes here: [Autosubmit Snippet](https://earth.bsc.es/gitlab/es/auto-s2s/-/snippets/113)
|
|
|
|
|
|
|
|
**Step 3**: On your workstation or on nord3v2, cd to the code directory and run:
|
|
|
|
```shell
|
|
|
|
source MODULES
|
|
|
|
Rscript split.R <path_to_your_recipe>
|
|
|
|
```
|
|
|
|
This will create an output directory, split your recipe into atomic recipes for independent verifications, and modify the configuration of your experiment according to the parameters in the recipe.
|
|
|
|
At the end of the script, some instructions will be written in the terminal explaining what you need to do to finish the set-up and start running your experiment. Follow them.
|
|
|
|
|
|
|
|
**Step 4**: You're done! You can and check the status of your experiment in the [Autosubmit GUI](https://earth.bsc.es/autosubmitapp/). If your experiment doesn't show up at first, try disabling the "Only Active" option next to the search bar.
|
|
|
|
|
|
|
|
## Recipes
|
|
|
|
|
|
|
|
Here is a complete example of a 'main' recipe that can be split for independent verifications.
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
################################################################################
|
|
|
|
## RECIPE DESCRIPTION
|
|
|
|
################################################################################
|
|
|
|
Description:
|
|
|
|
Author: V. Agudetse
|
|
|
|
Info: Example for recipe splitting
|
|
|
|
################################################################################
|
|
|
|
## ANALYSIS CONFIGURATION
|
|
|
|
################################################################################
|
|
|
|
Analysis:
|
|
|
|
Horizon: Seasonal
|
|
|
|
Variables: # A list of variables and their frequency
|
|
|
|
- {name: tas, freq: monthly_mean}
|
|
|
|
- {name: prlr, freq: monthly_mean}
|
|
|
|
Datasets:
|
|
|
|
System: # A list of systems
|
|
|
|
- {name: Meteo-France-System7}
|
|
|
|
- {name: ECMWF-SEAS5}
|
|
|
|
Multimodel: False # single option
|
|
|
|
Reference:
|
|
|
|
- {name: ERA5}
|
|
|
|
Time:
|
|
|
|
sdate: # A list of start dates
|
|
|
|
- '1101'
|
|
|
|
- '1201'
|
|
|
|
fcst_year: '2020'
|
|
|
|
hcst_start: '1993'
|
|
|
|
hcst_end: '2016'
|
|
|
|
ftime_min: 1
|
|
|
|
ftime_max: 6
|
|
|
|
Region: # A list of regions. If there's more than one region, they need to have a 'name'.
|
|
|
|
- {name: "global", latmin: -90, latmax: 90, lonmin: 0, lonmax: 359.9}
|
|
|
|
- {name: "nino34", latmin: -5, latmax: 5, lonmin: -10, lonmax: 60}
|
|
|
|
Regrid:
|
|
|
|
method: bilinear
|
|
|
|
type: to_system
|
|
|
|
Workflow:
|
|
|
|
Calibration:
|
|
|
|
method: mse_min
|
|
|
|
Skill:
|
|
|
|
metric: RPS, RPSS, CRPS, CRPSS, FRPSS, BSS10, BSS90, mean_bias, mean_bias_SS
|
|
|
|
Probabilities:
|
|
|
|
percentiles: [[1/3, 2/3], [1/10, 9/10]]
|
|
|
|
ncores: 7
|
|
|
|
remove_NAs: yes
|
|
|
|
Output_format: S2S4E
|
|
|
|
################################################################################
|
|
|
|
## Run CONFIGURATION
|
|
|
|
################################################################################
|
|
|
|
Run:
|
|
|
|
Loglevel: INFO
|
|
|
|
Terminal: yes
|
|
|
|
filesystem: esarchive
|
|
|
|
output_dir: /esarchive/scratch/vagudets/auto-s2s-outputs/ # replace with the directory where you want to save the outputs
|
|
|
|
code_dir: /esarchive/scratch/vagudets/repos/auto-s2s/ # replace with the directory where your code is
|
|
|
|
autosubmit: yes
|
|
|
|
# fill only if using autosubmit
|
|
|
|
auto_conf:
|
|
|
|
script: /esarchive/scratch/vagudets/repos/auto-s2s/modules/test_parallel_workflow.R # replace with the path to your script
|
|
|
|
expid: a5no # replace with your EXPID
|
|
|
|
hpc_user: bsc32762 # replace with your hpc username
|
|
|
|
wallclock: 01:00 # hh:mm
|
|
|
|
processors_per_job: 8
|
|
|
|
platform: nord3v2
|
|
|
|
email_notifications: yes # enable/disable email notifications. Change it if you want to.
|
|
|
|
email_address: victoria.agudetse@bsc.es # replace with your email address
|
|
|
|
notify_completed: no # notify me by email when a job finishes
|
|
|
|
notify_failed: yes # notify me by email when a job fails
|
|
|
|
``` |
