# Hands-on 4: Seasonal forecast verification

## Goal
Learn how to use the startR workflow to finish a piece of analysis, including defining and pre-processing the desired data,
defining the operation, building the workflow, and executing the operation.

In this use case, the ranked probability skill score (RPSS) and the root mean square error skill score (RMSSS) are computed to verify the forecast.
To make the process faster, the required data size is small here so we can run the execution on workstation.

## 0. Load required packages

```r
# Clean the session
rm(list = ls())

library(startR)
library(s2dv)
```

## 1. Define data
The first step of the analysis is use Start() to identify the data needed.

Hints:  
(1) The data paths use two '$' as the wildcard, and it can be given any names you like.   
(2) The variable we want to use is 'tas'.  
(3) The required time (sdate) period is November  1991 to 2011.  
(4) Read global data (i.e., full spatial grids.)  
(5) Take all the ensemble members.  
(6) Because we are going to use the whole startR workflow, we only need to create a pointer to the data repository rather than retrieve it to the workstation.

```r
# exp data
  # Use this one if on workstation or nord3 (have access to /esarchive)
  repos_exp <- paste0('/esarchive/exp/ecmwf/system5c3s/monthly_mean/',
                      '$var$_f6h/$var$_$sdate$.nc')
  # Use this one if on Marenostrum4 and log in with PATC2021 account
  repos_exp <- paste0('/gpfs/scratch/nct01/nct01127/d3_R_handson/esarchive/',
                      'exp/ecmwf/system5c3s/monthly_mean/',
                      '$var$_f6h/$var$_$sdate$.nc')

  sdates_exp <- sapply(1991:2011, function(x) paste0(x, '1101'))

  exp <- Start(dat = repos_exp,
               var = ___<fill>___,
               sdate = sdates_exp,
               time = indices(1),
               ensemble = ___<fill>___,
               latitude = ___<fill>___,
               latitude_reorder = Sort(),
               longitude = ___<fill>___,
               longitude_reorder = CircularSort(0, 360),
               synonims = list(longitude = c('lon', 'longitude'),
                               latitude = c('lat', 'latitude')),
               return_vars = list(latitude = NULL, longitude = NULL, time = 'sdate'),
               retrieve = FALSE)

# obs data
  # Use this one if on workstation or nord3 (have access to /esarchive)
  repos_obs <- paste0('/esarchive/recon/ecmwf/era5/monthly_mean/$var$_f1h-r360x180/',
                      '$var$_$sdate$.nc')
  # Use this one if on Marenostrum4 and log in with PATC2022 account
  repos_obs <- paste0('/gpfs/scratch/nct01/nct01127/d3_R_handson/esarchive/',
                      'recon/ecmwf/era5/monthly_mean/',
                      '$var$_f1h-r360x180/$var$_$sdate$.nc')

  sdates_obs <- sapply(1991:2011, function(x) paste0(x, '11'))

  obs <- Start(dat = repos_obs,
               var = ___<fill>___,
               sdate = sdates_obs,
               time = indices(1),
               latitude = ___<fill>___,
               latitude_reorder = Sort(),
               longitude = ___<fill>___,
               longitude_reorder = CircularSort(0, 360),
               synonims = list(longitude = c('lon', 'longitude'),
                               latitude = c('lat', 'latitude')),
               return_vars = list(latitude = NULL, longitude = NULL, time = 'sdate'),
               retrieve = FALSE)
```

Question:  

1. What are the dimensions of these two data?

2. What is the size of these two data?

3. Check the latitude, longitude, and time values of the two data. Are they consistent?


## 2. Define operation and workflow
It is recommended to define the function and write Step() together, because the latter one helps you clarify the input and output dimensions of the function.

In the self-defined function, we want to use the two functions, 's2dv::RPSS' and 's2dv::RMSSS', to calculate the skill scores.
You can check the functions by typing `?s2dv::RPSS` and `?s2dv::RMSSS`, or on [s2dv GitLab](https://earth.bsc.es/gitlab/es/s2dv/-/blob/master/R/).

Hint:
(1) The self-defined function is for exp and obs data. Therefore, target_dims and output_dims in Step() should be a list with two vector elements.  
(2) The essential dimensions are "ensemble" and "sdate" for `exp`, and "sdate" for `obs`. These dimensions should be the 'target_dims' in Step().  
(3) To return the two skill scores as well as the significance test results, put them in a list of 4 elements.  
(4) What are the dimensions of the outputs? These dimensions should be 'output_dims' in Step().  
(5) The first input of AddStep() should also be a list containing exp and obs.  

```r
  # self-defined function
  skill_func <- function(exp, obs) {
    # exp: [ensemble, sdate]
    # obs: [sdate]

    # Calculate RPSS
    rpss <- s2dv::RPSS(exp, obs, memb_dim = ___<fill>___)

    # Calculate RMSSS
    ## ensemble mean
    exp_ens <- s2dv::MeanDims(exp, ___<fill>___, na.rm = T)
    rmsss <- s2dv::RMSSS(exp_ens, obs, time_dim = ___<fill>___, dat_dim = NULL)

    return(list(rpss = rpss$rpss, rpss_sign = rpss$sign, 
                rmsss = rmsss$rmsss, rmsss_pval = rmsss$p.val))
  }
  step <- Step(skill_func, target_dims = list(exp = ___<fill>___, obs = ___<fill>___),
               output_dims = list(rpss = NULL, rpss_sign = NULL, rmsss = NULL, rmsss_pval = NULL))
  wf <- AddStep(list(exp, obs), step)
```

Question:  
1. Which dimensions are used in operation? Which dimensions are free?

2. Can we use more dimensions as target_dims? What are the disadvantages?  


## 3. Execute locally
To avoid potential technical problems in the connection and configuration, 
we choose to run the execution locally. 
Noted that it is recommended to submit jobs to HPCs if the data size is big or 
the operation is heavy.

Hint:  
(1) Use the free dimensions (i.e., those are not 'target_dims' in Step()) to chunk.  
(2) It is safe to divide the data into pieces of which the size each is 1/2-1/3 of RAM. In this case, 
the data size is only around 270Mb, so you can play with it without worrying about crashing the terminal.  

```r
  res <- Compute(wf$rpss,
                 chunks = list(___<fill>___),
                 threads_load = 2,
                 threads_compute = 4)
```

## 4. Check the results

1. What is the Compute summary printed on the screen? Do you see the total time, time for load, time for compute?

2. Check the list structure using function `str()`. What are the items in the list?

3. What are the dimensions of each item?

4. Plot the maps

Use `s2dv::PlotEquiMap` or other visualization tools to plot the maps. Check the usage of PlotEquiMap() on s2dv GitLab or type `?s2dv::PlotEquiMap`.  

```r
  # Get latitude and longitude from the attributes
  lat <- as.vector(attr(exp, 'Variables')$common$latitude)
  lon <- as.vector(attr(exp, 'Variables')$common$longitude)

  # Set the color bars: check data range to decide the appropriate color bar
  print(range(res$rpss, na.rm = TRUE))
  brks_rpss <- seq(___<fill>___, ___<fill>___, by = ___<fill>___)
  print(range(res$rmsss, na.rm = TRUE))
  brks_rmsss <- seq(___<fill>___, ___<fill>___, by = ___<fill>___)
 
  # Plot RPSS
  ##NOTE: Remove 'fileout' to view the figure in pop-up window
  PlotEquiMap(___<fill>___, lon = lon, lat = lat,
              dots = ___<fill>___, dot_size = 1.5, dot_symbol = 3,
              brks = brks_rpss,
              filled.continents = FALSE, triangle_ends = c(TRUE, TRUE),
              toptitle = 'ECMWF system5 monthly mean tas RPSS 1991-2011', title_scale = 0.6,
              fileout = '~/handson_4_fig_rpss.png')

  # Plot RMSSS
  PlotEquiMap(___<fill>___, lon = lon, lat = lat,
              color_fun = clim.palette('yellowred'), 
              brks = brks_rmsss,
              filled.continents = FALSE, triangle_ends = c(TRUE, TRUE),
              toptitle = 'ECMWF system5 monthly mean tas RMSSS 1991-2011', title_scale = 0.6,
              fileout = '~/handson_4_fig_rmsss.png')
```


**BONUS**: Try to use only one thread for loading and computing (i.e., `threads_load = 1`, `threads_compute = 1`.) Is it slower than using more threads?
