# Hands-on 1.2: Computation of Scorecards with Autosubmit

Author: An-Chi Ho

## Goal
Compute some skill metrics and plots scorecards with SUNSET, using Autosubmit to dispatch jobs in parallel.
In the recipe, we request 12 start dates (0101, 0201, ..., 1201). SUNSET will split the recipe into 12 atomic recipes, and Autosubmit will run 12 jobs, which process the verification, for each recipe in parallel.
When all the verification jobs are finished, the scorecard job will be triggered and produces the scorecards.

We only use one variable, one model and one reference dataset in this example, but you can add more datasets and variables if needed, and SUNSET will split the recipes accordingly. 

Check GitLab Wiki:
- Autosubmit page for full explanation of using SUNSET with Autosubmit https://earth.bsc.es/gitlab/es/sunset/-/wikis/Autosubmit

- Home page Scorecards module section to know more about scorecards https://earth.bsc.es/gitlab/es/sunset/-/wikis/home#scorecards-module


## 0. Cloning the SUNSET repository

If you're completely new to SUNSET, the first step is to create a copy of the tool in your local environment. 
Open a terminal and `cd` to the directory where you would like to store your local copy of SUNSET. For example: `/esarchive/scratch/<your_username>/git/`. If a directory does not exist yet, you can create it with the `mkdir` shell command.

```shell
# Clone the GitLab repository to create a local copy of the code
git clone https://earth.bsc.es/gitlab/es/sunset.git
```
You should see a git folder "sunset" under the current directory. Now you have all the code, recipes, and scripts for running SUNSET.


## 1. Create Autosubmit experiment

Since we're going to use Autosubmit to dispatch jobs, we need to have an Autosubmit experiment. Note that SUNSET uses Autosubmit >= 4.0.0.

On the workstation or the Autosubmit machine, you can create an experiment by the following commands.

```shell
module load autosubmit/4.0.0b-foss-2015a-Python-3.7.3
autosubmit expid -H nord3v2 -d "SUNSET use case 1_2" 
```
You will see the messages like below:

```shell
Autosubmit is running with 4.0.0b
The new experiment "a6pc" has been registered.
Generating folder structure...
Experiment folder: /esarchive/autosubmit/a6pc
Generating config files...
Experiment a6pc created
```
Note the experiment ID (in this snippet above, a6pc) down. We need it for the recipe later.


## 2. Modifying the recipe

The template recipe for this use case can be found in [ex1_2-recipe.yml](use_cases/ex1_2_autosubmit_scorecards/ex1_2-recipe.yml). 
You should at least edit some items in the "Run" section:
- `output_dir`: The directory you want to save the outputs and logs
- `code_dir`: The directory where your SUNSET code is stored (i.e., the git folder)
- `auto_conf$script`: The path to the script ex1_2-recipe.yml
- `auto_conf$expid`: The experiment "xxxx" you just created
- `auto_conf$hpc_user`: You user ID on Nord3, which should be bsc032xxx
- `auto_conf$email_address`: Your email. You can also adjust other email notification parts up to your preference.

In the recipe, we ask for anomaly calculation after loading the data, calculate the skill scores and save the result for scorecards. In the Scorecard section, three regions are requested.

Feel free to also modify other aspects according to your particular needs. You can read more about the parameters and the available modules in the SUNSET GitLab wiki.

## 3. The user-defined script

We need to have a script to define the modules to use and the steps of the workflow. Note that the script is for data loading and verification parts. The Scorecards module doesn't need to be included in this script.

The prepare_outputs() function is already incorporated into the launcher script (see the next section for details about launcher), so it does not need to be included in the user-defined script in this case. 
In its place, we will use the function read_atomic_recipe(). The recipe path is passed as an argument onto the R script. Below is an example of what the script should look like. 

NOTE: Don't try to copy and paste this code on an R terminal; it will not work without properly pre-processing the recipe and creating the necessary output folders first.

The beginning of the user-defined script should look like this:

```R
# Load modules
source("modules/Loading/Loading.R")
source("modules/Anomalies/Anomalies.R")
source("modules/Skill/Skill.R")
source("modules/Saving/Saving.R")

# Read recipe 
## (leave this part as-is! Autosubmit will automatically pass the atomic recipe as a parameter)
args = commandArgs(trailingOnly = TRUE)
recipe_file <- args[1]
recipe <- read_atomic_recipe(recipe_file)
```

The rest of the user-defined script can be written in the same way as any other SUNSET script. We load the data, calculate the anomalies, then compute the skill scores and save the result as netCDF files for Scorecards.

```R
# Load data
data <- Loading(recipe)
# Compute tos anomalies
data <- Anomalies(recipe, data)
# Compute skill metrics
skill_metrics <- Skill(recipe, data)
```
Check the example script at [ex1_2-script.yml](use_cases/ex1_2_autosubmit_scorecards/ex1_2-script.R).
You can execute it as-is or copy it and modify it according to your needs.


## 4. Launch jobs and Use Autosubmit

We will start the jobs with the launcher. The SUNSET Launcher is a bash script named launch_SUNSET.sh that can be found in the main directory of the SUNSET repository. It runs in two steps:

1. Run the recipe checks, split the recipe into atomic recipes and create the directory for the outputs.
2. Modify the Autosubmit configuration of your experiment according to the parameters in the recipe.

The bash script needs two inputs: (1) [recipe](#2-modifying-the-recipe) (2) [R script](#3-the-user-defined-script).

 On your workstation or Nord3 under the SUNSET code directory, run:

```shell
bash launch_SUNSET.sh use_cases/ex1_2_autosubmit_scorecards/ex1_2-recipe.yml use_cases/ex1_2_autosubmit_scorecards/ex1_2-script.R 
```
You will see the messages similar to below:
```shell
[1] "Saving all outputs to:"
[1] "/esarchive/scratch/aho/auto-s2s-outputs/ex1_2-recipe_20231129003740"
INFO  [2023-11-29 00:37:41] Checking recipe: use_cases/ex1_2_autosubmit_scorecards/ex1_2-recipe.yml
WARN  [2023-11-29 00:37:41] The element 'fcst_year' is not defined in the recipe. No forecast year will be used.
INFO  [2023-11-29 00:37:41] ##### RECIPE CHECK SUCCESSFULL #####
INFO  [2023-11-29 00:37:41] Splitting recipe in single verifications.
INFO  [2023-11-29 00:37:41] The main recipe has been divided into 12 atomic recipes.
INFO  [2023-11-29 00:37:41] Check output directory /esarchive/scratch/aho/auto-s2s-outputs//ex1_2-recipe_20231129003740/logs/recipes/ to see all the individual atomic recipes.
INFO  [2023-11-29 00:37:41] ##### AUTOSUBMIT CONFIGURATION WRITTEN FOR a6pc #####
INFO  [2023-11-29 00:37:41] You can check your experiment configuration at: /esarchive/autosubmit/a6pc/conf/
INFO  [2023-11-29 00:37:41] Please SSH into bscesautosubmit01 or bscesautosubmit02 and run the following commands:
INFO  [2023-11-29 00:37:41] module load autosubmit/4.0.0b-foss-2015a-Python-3.7.3
INFO  [2023-11-29 00:37:41] autosubmit create a6pc
INFO  [2023-11-29 00:37:41] autosubmit refresh a6pc
INFO  [2023-11-29 00:37:41] nohup autosubmit run a6pc & disown
```
You can see some useful information, like the the path to atomic recipes, the Autosubmit configuration files, and most importantly, follow the last lines to launch your experiment. 

```shell
ssh bscesautosubmit01.bsc.es
(enter Autosubmit machine)
module load autosubmit/4.0.0b-foss-2015a-Python-3.7.3
autosubmit create a6pc
autosubmit refresh a6pc
nohup autosubmit run a6pc & disown
```

Then, you can go to [Autosubmit GUI](https://earth.bsc.es/autosubmitapp/) to check the experiment status.

<img src="Figures/as_tree.PNG" width="800" /> 

As you can see, the Scorecards job is dependent on the Verification jobs. Once the 12 verification jobs are finished, the Scorecards job will start.

## 5. Results and plots

The scorecards are saved under `plots/Scorecards` under the output directory. There will be 4 files (_more explanation here_)

<img src="Figures/scorecard-1_ECMWF-SEAS5_ERA5_tas_1993-2003.png" width="800" /> 
<img src="Figures/scorecard-2_ECMWF-SEAS5_ERA5_tas_1993-2003.png" width="800" /> 
<img src="Figures/scorecard-3_ECMWF-SEAS5_ERA5_tas_1993-2003.png" width="800" /> 
<img src="Figures/scorecard-4_ECMWF-SEAS5_ERA5_tas_1993-2003.png" width="800" /> 


## 6. Rerun Autosubmit

If something goes wrong and makes the jobs fail, you can rerun the failed jobs only. 

1. Go to Autosubmit GUI, select the failed job(s), click "CHANGE STATUS"
2. Select "Set status to:" as "WAITING". Copy the lines and run them on Autosubmit machine or workstation.
3. Fix the problem under your local SUNSET git directory.
4. Run `autosubmit refresh xxxx` and `nohup autosubmit run xxxx & disown`.

<img src="Figures/as_change_status.PNG" width="600" /> 

If everything fails, you can also simply recreate the experiment by `autosubmit create xxxx` --> `autosubmit refresh xxxx` --> `nohup autosubmit run xxxx & disown`.

