# Hands-on 1.3: Computation of El Niño indices for two seasonal models

Author: Victòria Agudetse Roures and Núria Pérez Zanón

## Goal
Create a SUNSET recipe to compute and evaluate the skill of several El Niño indices (Niño1+2, Niño3, Niño3.4 and Niño4) for two models: ECMWF-SEAS5 and MeteoFrance System 7. We include the information for both of the models in a single recipe, and the SUNSET Launcher will split the recipe into two 'atomic recipes': one for each model. The computation for atomic recipe will be run in the cluster as two separate jobs.

It is also possible to split a recipe along different Reference datasets, Variables and Start Dates.

## 0. Cloning the SUNSET repository

The first step to use SUNSET is to create a copy of the code in your local environment. Open a terminal and `cd` to the directory where you would like to store your local copy of SUNSET. For example: `/esarchive/scratch/<your_username>/git/`. If a directory does not exist yet, you can create it with the `mkdir` shell command.

```shell
# Clone the GitLab repository to create a local copy of the code
git clone https://earth.bsc.es/gitlab/es/sunset.git
```

## 1. Modifying the recipe

The template recipe for this use case can be found in `use_cases/ex1_3_nino_indices_comparison/ex1_3-recipe.yml`. You should open it with an editor such as emacs or vim:

```shell
# cd to the main SUNSET directory
# Open the recipe with a text editor such as vim or emacs
vim use_cases/ex1_3_nino_indices_comparison/ex1_3-recipe.yml
```

Then, under the 'Run' section of the recipe, you should edit the parameters `output_dir` and `code_dir` to point to your desire output directory and to the directory where your SUNSET code is stored, respectively.

Feel free to also modify other aspects of the reicpe according to your particular needs. You can read more about the parameters and the available modules in the SUNSET wiki.

## 2. The user-defined script

The SUNSET Launcher is a bash script named launch_SUNSET.sh that can be found in the main directory of the SUNSET repository. When working without Autosubmit, it runs in two steps:

1. Running the recipe checks, splitting the recipe into atomic recipes and creating the directory for the outputs.
2. Sending jobs to the cluster to run the user-defined script for each atomic recipe, using SLURM.

The `prepare_outputs()` function is already incorporated into the first step. For that reason, it does not need to be included in the user-defined script in this particular case. In its place, we will use the function `read_atomic_recipe()`. The recipe path is passed as an argument onto the R script. Below is an example of what the script should look like.

> :information_source: **NOTE**: Don't try to copy and paste this code on an R terminal; it will not work without properly pre-processing
 the recipe and creating the necessary output folders first. See [Steps To Use SUNSET](https://docs.google.com/presentation/d/1FdW25b4JZLxXDkVRVIOz-P_2A1WXLWDXhxZvisrVdqU/edit#slide=id.g308472768fe_0_124) for more details.

The beginning of the user-defined script should look like this:

```R
# Load the modules to be used
source("modules/Loading/Loading.R")
source("modules/Units/Units.R")
source("modules/Anomalies/Anomalies.R")
source("modules/Indices/Indices.R")
source("modules/Skill/Skill.R")

# Define the recipe path as the first argument from the command line
args = commandArgs(trailingOnly = TRUE)
recipe_file <- args[1]
# Read the atomic recipe
recipe <- read_atomic_recipe(recipe_file)
```

The rest of the user-defined script can be written in the same way as any other SUNSET script:

```R
# Load data
data <- Loading(recipe)
# Check units and transform if needed
data <- Units(recipe, data)
# Compute tos anomalies
data <- Anomalies(recipe, data)
# Compute Niño Indices
nino_indices <- Indices(data = data, recipe = recipe)

# We can compute the Skill metrics for each of the El Niño indices,
# specifying that the data is spatially aggregated, with the parameter
# agg = "region".
for (index in nino_indices) {
  nino_skill_metrics <- Skill(recipe = recipe, data = index, agg = "region")
}
```

A complete, ready-to-use sample of this example script can be found in `use_cases/ex1_3_nino_indices_comparison/ex1_3-script.R`. You can execute it as-is or copy it and modify it according to your specific needs.

## 3. Launching the jobs with the SUNSET Launcher

The first step is to connect to the HPC machine through `ssh`. When working without Autosubmit, the SUNSET Launcher should be run directly from the HPC machine where the jobs will run (for example, Nord4). There is no need to request an interactive session; the launcher script can be called directly from the login node. You can obtain detailed usage information by running:

```shell
bash launch_SUNSET.sh --help
```

The mandatory arguments are the paths to the recipe and the script. We can also include other optional arguments to be used by SLURM, such as the number of CPUs to request (--cpus), the wallclock time for each job (--wallclock) and other extra directives (--custom_directives). You can refer to the [Nord4 user guide](https://www.bsc.es/supportkc/docs/Nord4/slurm#job-directives) and the [SLURM sbatch documentation](https://slurm.schedmd.com/sbatch.html) for more information on the available options for the parameters.

In this case, we are giving each job a wallclock time of 1 hour and requesting exclusive usage of all the cores in one node. The shell command to run SUNSET will look like this:

```shell
bash launch_SUNSET.sh use_cases/ex1_3_nino_indices_comparison/ex1_3-recipe.yml use_cases/ex1_3_nino_indices_comparison/ex1_3-script.R --wallclock=01:00:00 --custom_directives="--exclusive"
```

You can check the status of your running jobs with the `squeue` command. The SLURM logs will be inside of your code directory, in a subfolder named 'out-logs'. It can be useful to check them in case of errors.

## 4. Results and plots

The spatial pattern and time series plots that were requested are saved inside `plots/Indices/` in the output directory. There will be one set of plots for each El Niño index, with a descriptive file name providing information about the content of the plot, the system/reference datasets, the start date and the forecast time. Here are some examples of the results:

Spatial correlation for the ensemble mean:

![](./figures/nino34_correlation_tos_ensmean_ECMWF-SEAS5_s1101_ftime02.png)
![](./figures/nino34_correlation_tos_ensmean_Meteo-France-System7_s1101_ftime02.png)

Time series comparison between the model and the reference dataset (ERA5):
![](./figures/nino34_ECMWF-SEAS5_ERA5_s1101_ftime02.png)
![](./figures/nino34_Meteo-France-System7_ERA5_s1101_ftime02.png)
