# Hands-on 1.1: Single Verification Workflow on the Terminal

Author: Victòria Agudetse Roures

## Goal
Create a SUNSET recipe and use the functions in the suite to reproduce the verification workflow from the previous hands-on exercises.

## 0. Cloning the SUNSET repository

The first step to use SUNSET is to create a copy of the code in your local environment. Open a terminal and `cd` to the directory where you would like to store your local copy of SUNSET. For example: `/esarchive/scratch/<your_username>/git/`. If a directory does not exist yet, you can create it with the `mkdir` shell command.

```shell
# Clone the GitLab repository to create a local copy of the code
git clone https://earth.bsc.es/gitlab/es/sunset.git
```

## 1. Modifying the recipe

SUNSET uses YAML configuration files called 'recipes' to specify which data you want to load and the details of the different steps of the workflow. In this example, we want to evaluate the temperature-at-surface (tas) monthly means, using MeteoFrance System 7 data as our experiment and ERA5 as our reference dataset, for the initialization month of November.

There is a template file for this hands-on tutorial, which you can open with a text editor:

```shell
# cd to the main SUNSET directory
# Open the recipe with a text editor such as vim or emacs
vim use_cases/ex1_1_single_analysis_terminal/ex1_1-recipe.yml
```

Once you have opened your recipe, it is time to edit the contents. Fill in the blank slots according to your preference, based on the options given in the description. 

```yaml
Description:
 Author: <___>
 Description: Exercise 1.1: Calibration and skill assessment of MeteoFrance System 7 surface temperature
Analysis:
 Horizon: Seasonal
 Variables:
   name: tas
   freq: monthly_mean
   units: <___> # Choose your units: C or K
 Datasets:
   System:
     name: Meteo-France-System7
     Multimodel: no
   Reference:
     name:  ERA5
 Time:
   sdate: '1101'
   fcst_year: '2020'
   hcst_start: '1993'
   hcst_end: '2016'
   ftime_min: <___> # Choose the first time step! A number from 1 to 6
   ftime_max: <___> # Choose the last time step! A number from 1 to 6
 Region:
   name:  "EU"
   latmin:  20
   latmax:  80
   lonmin:  -20
   lonmax:  40
 Regrid:
   method:  bilinear
   type:  'r360x181' # options: to_system, to_reference, self-defined grid
 Workflow:
   Anomalies:
     compute: yes
     cross_validation: yes
     save: 'none'
   Calibration:
     method: <___>
     save: 'none'
   Skill:
     metric: RPSS,  BSS10,  BSS90
     cross_validation: yes
     save: 'none'
   Probabilities:
     percentiles:  [[1/3,  2/3],  [1/10,  9/10]]
     save: 'none'
   Visualization:
     plots: skill_metrics,  forecast_ensemble_mean,  most_likely_terciles
     multi_panel: no
     projection: cylindrical_equidistant
   ncores: 10
   remove_NAs: yes
   Output_format: S2S4E
Run:
 Loglevel: INFO
 Terminal: yes
 output_dir: <______> # Path to the directory where you want your outputs to be saved
 code_dir: <______> # Path to the directory where your code is
```

## 2. Load the required SUNSET modules and read the recipe

First of all, source the MODULES file to load the environment modules needed to run SUNSET.

```shell
source MODULES
```

Open an R session, by simply typing `R` on the terminal.

To run SUNSET, we must run the R session from the directory where the code is. To check your working directory, you can run the shell command `pwd`. From the R session, you can use the commands `getwd()` and `setwd()` to see and change your working directory.

```r
# Load required modules
source("modules/Loading/Loading.R")
source("modules/Units/Units.R")
source("modules/Calibration/Calibration.R")
source("modules/Anomalies/Anomalies.R")
source("modules/Skill/Skill.R")
source("modules/Saving/Saving.R")
source("modules/Visualization/Visualization.R")

# Read recipe
recipe_file <- "use_cases/ex1_1_single_analysis_terminal/ex1_1-recipe.yml"
recipe <- prepare_outputs(recipe_file)
```

The function `prepare_outputs()` creates a unique folder for the logs, data files and plots that result from the execution of your recipe, inside the directory you specified.  It also runs a check over the recipe to detect any potential errors, misspellings or missing arguments. At the end of the check, a message is displayed indicating whether or not the recipe passed the check, along with the list of errors and warnings.

**Questions**

Read the logs!

(1) Did your recipe pass the check? Did you get any warnings?

(2) Where will your outputs be saved? Copy and paste this directory somewhere, so that you can check it later!

*Tip*: The recipe is now stored as a `list` containing all the information of the original YAML file, plus some extra things! If you want to see any particular element of the recipe from the R session, you can simply access that element in the list. For example: 

```r
# Checking the variable name
recipe$Analysis$Variables$name
# Checking the output directory
recipe$Run$output_dir
```

## 3. Load the data and change the units

The **Loading** module retrieves the information from the recipe to load the data that has been requested it in. It loads the experiment data for the hindcast period, the reference data for the corresponding period, and the experiment forecast if a forecast year has been requested. 

For certain variables like temperature, precipitation or sea level pressure, the user can request for specific units to load the data in. The **Units** module will read the original units as stored in the netCDF files and perform any necessary unit converstions to match the request in the recipe. It also verifies that all of the loaded datasets share the same units, even if no specific unit has been requested. For this reason, users are strongly encouraged to run it even if they did not request any unit conversion.

```r
# Load datasets
data <- Loading(recipe)
# Change units
data <- Units(recipe, data)
```

**Questions**

(1) What is the structure of `data`? What is the class of the objects in `data`? *Tip*: you can use functions like `class()`, `names()` or `str()` to gain information about the structure of the object and its contents.

```r
class(data)
names(data)
str(data, max.level = 2)
# You can access any of the three objects with the `$` operator:
class(data$hcst)
```

(2) Pay attention to the log messages: Did your units get converted? Are the new units what you expect? You can check the metadata of any of the objects in data. SUNSET also provides the `data_summary()` function, which lets you have a quick look at your objects:

```r
# Check the new units and data of hcst, fcst and/or obs. Are they all the same?
data$hcst$attrs$Variable$metadata$tas$units
data_summary(data$hcst, recipe)
data_summary(data$fcst, recipe)
data_summary(data$obs, recipe)
```
(3) What are the dimensions of the datasets? Are they consistent with what is requested in the recipe? *Tip*: Check the data summary!

## 4. Calibrate the data and compute the anomalies

SUNSET has a few modules to perform post-processing on the experimental and the reference datasets. The **Calibration** module performs the bias correction method indicated in the recipe, using the `CSTools::CST_Calibration()` function. 

The **Anomalies** module removes the climatologies using functions like `CSTools::CST_Anomaly()` and `s2dv::Clim()`, and also returns the full fields in case they are needed for any future computations.

```r
# Calibrate the data
data <- Calibration(recipe, data)
# Compute anomalies
data <- Anomalies(recipe, data)
```
**Questions**

(1) Verify that you now have anomaly values instead of the original full field. *Tip*: Use `data_summary()` like in the previous example and pay attention to the new values.

## 5. Evaluate the model skill and compute the probability thresholds

The **Skill** module returns a list of all the evaluation metrics requested in the recipe, in the shape of multi-dimensional arrays. In this case, we will compute three metrics:

- **RPSS (Ranked Probability Skill Score)**: This skill score measures how well a forecast predicts the probability of the tercile categories (below normal, normal and above-normal), compared to the climatology.
- **BSS10 and BSS90 (Brier Skill Score):** This skill score measures how well a forecast predicts the probability of the 10th percentile and 90th percentile extreme events, compared to the climatology.

The `Probabilities()` function returns the probability values for each requested category for the hindcast and forecast data, as well as the hindcast percentile values corresponding to each threshold.
```
# Compute skill metrics
skill_metrics <- Skill(recipe, data)
# Compute percentiles and probability bins
probabilities <- Probabilities(recipe, data)
```
**Questions**

(1) What is the structure of `skill_metrics`? Which metrics were computed? What dimensions do they have? *Tip*: use `str()` and `names()`.

(2) What is the structure of `probabilities`? Can you identify the probability categories and the percentiles? *Tip*: use `str()`.

## 6. Plotting the results

Now, let's visualize the information that was computed!

The **Visualization** module will generate the three types of maps we requested in the recipe: 
- Skill maps to visualize the skill distribution of the model, for each metric.
- The ensemble mean of the calibrated forecast anomalies.
- A map showing the most likely tercile category for each point in the grid.

With the significance option in the `Visualization()` function, you can choose whether or not to shade the grid points that are statistically significant in each skill metric plot.

```r
# Plot data
Visualization(recipe, data, 
              skill_metrics = skill_metrics,
              probabilities = probabilities,
              significance = TRUE)
```

Now, you can `cd` to the the output directory and inspect the contents of the `plots/` subdirectory. The plots are png files that can be visualized with the `display` command. They have a descriptive name including the content of the plot, the date and the forecast time.

**Questions**

(1) Let's take a look at the forecast ensemble mean. What is the sign of the anomalies over Spain? In what regions are the anomalous temperatures strongest?

(2) Let's take a look at the skill metrics RPSS, BSS10 and BSS90. In what regions and for which metrics is the forecast most skillful? *Tip*: Positive values indicate that the model is a better predictor than the climatology, with 1 being the perfect score.

(3) Let's take a look at the Most Likely Terciles plots. This plot indicates the probability of the temperature being below normal, near-normal or above normal. What is the most likely category for Spain?
