# Hands-on 4: Seasonal forecast verification

## Goal
Learn how to use the startR workflow to finish a piece of analysis, including defining and pre-processing the desired data,
defining the operation, building the workflow, and executing the operation.

In this use case, the ranked probability skill score (RPSS) and the root mean square error skill score (RMSSS) are computed to verify the forecast.
To make the process faster, the required data size is small here so we can run the execution on workstation.

## 0. Load required packages

```r
# Clean the session
rm(list = ls())

library(startR)
library(s2dv)
```

## 1. Define data
The first step of the analysis is use Start() to identify the data needed.

Hints:  
(1) The data paths use two '$' as the wildcard, and it can be given any names you like.  
(2) The variable we want to use is 'tas'.   
(3) The required time (sdate) period is November 1991 to 2011.  
(4) Read global data (i.e., full spatial grids.) 
(5) Take all the ensemble members.                    
(6) Because we are going to use the whole startR workflow, we only need to create a pointer to the data repository rather than retrieve it to the workstation. 

```r
# exp data
  # Use this one if on workstation or nord3 (have access to /esarchive)
  repos_exp <- paste0('/esarchive/exp/ecmwf/system5c3s/monthly_mean/',
                      '$var$_f6h/$var$_$sdate$.nc')
  # Use this one if on Marenostrum4 and log in with PATC2021 account
  repos_exp <- paste0('/gpfs/scratch/nct01/nct01127/d3_R_handson/esarchive/',
                      'exp/ecmwf/system5c3s/monthly_mean/',
                      '$var$_f6h/$var$_$sdate$.nc')

  sdates_exp <- sapply(1991:2011, function(x) paste0(x, '1101'))

  exp <- Start(dat = repos_exp,
               var = 'tas',
               sdate = sdates_exp,
               time = indices(1),
               ensemble = 'all',
               latitude = 'all',
               latitude_reorder = Sort(),
               longitude = 'all',
               longitude_reorder = CircularSort(0, 360),
               synonims = list(longitude = c('lon', 'longitude'),
                               latitude = c('lat', 'latitude')),
               return_vars = list(latitude = NULL, longitude = NULL, time = 'sdate'),
               retrieve = FALSE)

# obs data
  # Use this one if on workstation or nord3 (have access to /esarchive)
  repos_obs <- paste0('/esarchive/recon/ecmwf/era5/monthly_mean/$var$_f1h-r360x180/',
                      '$var$_$sdate$.nc')
  # Use this one if on Marenostrum4 and log in with PATC2022 account
  repos_obs <- paste0('/gpfs/scratch/nct01/nct01127/d3_R_handson/esarchive/',
                      'recon/ecmwf/era5/monthly_mean/',
                      '$var$_f1h-r360x180/$var$_$sdate$.nc')

  sdates_obs <- sapply(1991:2011, function(x) paste0(x, '11'))

  obs <- Start(dat = repos_obs,
               var = 'tas',
               sdate = sdates_obs,
               time = indices(1),
               latitude = 'all',
               latitude_reorder = Sort(),
               longitude = 'all',
               longitude_reorder = CircularSort(0, 360),
               synonims = list(longitude = c('lon', 'longitude'),
                               latitude = c('lat', 'latitude')),
               return_vars = list(latitude = NULL, longitude = NULL, time = 'sdate'),
               retrieve = FALSE)
```

Question:  

1. What are the dimensions of these two data?  
```r
attr(exp, 'Dimensions')
      dat       var     sdate      time  ensemble  latitude longitude 
        1         1        21         1        25       181       360 

attr(obs, 'Dimensions')
      dat       var     sdate      time  latitude longitude 
        1         1        21         1       181       360 
```

2. What is the size of these two data?  
exp: 261Mb; obs: 10.4Mb as shown on screen from Start() call.

3. Check the latitude, longitude, and time values of the two data. Are they consistent?

```r
# latitude
as.vector(attr(exp, 'Variables')$common$lat)
as.vector(attr(obs, 'Variables')$common$lat)
# longitude
as.vector(attr(exp, 'Variables')$common$lon)
as.vector(attr(obs, 'Variables')$common$lon)
# time
attr(exp, 'Variables')$common$time
attr(obs, 'Variables')$common$time
```

## 2. Define operation and workflow
It is recommended to define the function and write Step() together, because the latter one helps you clarify the input and output dimensions of the function.

In the self-defined function, we want to use the two functions, 's2dv::RPSS' and 's2dv::RMSSS', to calculate the skill scores.
You can check the functions by typing `?s2dv::RPSS` and `?s2dv::RMSSS`, or on [s2dv GitLab](https://earth.bsc.es/gitlab/es/s2dv/-/blob/master/R/).  

Hint:  
(1) The self-defined function is for exp and obs data. Therefore, target_dims and output_dims in Step() should be a list with two vector elements.
(2) The essential dimensions are "ensemble" and "sdate" for `exp`, and "sdate" for `obs`. These dimensions should be the 'target_dims' in Step().  
(3) To return the two skill scores as well as the significance test results, put them in a list of 4 elements.  
(4) What are the dimensions of the outputs? These dimensions should be 'output_dims' in Step(). 
(5) The first input of AddStep() should also be a list containing exp and obs.

```r
  # self-defined function
  skill_func <- function(exp, obs) {
    # exp: [ensemble, sdate]
    # obs: [sdate]

    # Calculate RPSS
    rpss <- s2dv::RPSS(exp, obs, memb_dim = 'ensemble')

    # Calculate RMSSS
    ## ensemble mean
    exp_ens <- s2dv::MeanDims(exp, 'ensemble', na.rm = T)
    rmsss <- s2dv::RMSSS(exp_ens, obs, time_dim = 'sdate', dat_dim = NULL)

    return(list(rpss = rpss$rpss, rpss_sign = rpss$sign, 
                rmsss = rmsss$rmsss, rmsss_pval = rmsss$p.val))
  }
  step <- Step(skill_func, target_dims = list(exp = c('ensemble', 'sdate'), obs = c('sdate')),
               output_dims = list(rpss = NULL, rpss_sign = NULL, rmsss = NULL, rmsss_pval = NULL))
  wf <- AddStep(list(exp, obs), step)
```

Question:  
1. Which dimensions are used in operation? Which dimensions are free?  
'ensemble' and 'sdate' are used in the self-defined function. The other dimensions (dat, var, time, latitude, and longitude) 
are not necessary for the function. These free dimensions will be looped by multiApply during the execution.

2. Can we use more dimensions as target_dims? What are the disadvantages?  
It still works if putting more dimensions as target_dims, but we will lose the choices for chunking in the next step. Also, the lighter the function is, the quicker the operation runs.  


## 3. Execute locally
To avoid potential technical problems in the connection and configuration,
we choose to run the execution locally.
Noted that it is recommended to submit jobs to HPCs if the data size is big or
the operation is heavy.

Hint:  
(1) Use the free dimensions (i.e., those are not 'target_dims' in Step()) to chunk.  
(2) It is safe to divide the data into pieces of which the size each is 1/2-1/3 of RAM. In this case, 
the data size is only around 270Mb, so you can play with it without worrying about crashing the terminal.  

```r
  res <- Compute(wf$rpss,
                 chunks = list(latitude = 2, longitude = 2),
                 threads_load = 2,
                 threads_compute = 4)
```

## 4. Check the results

1. What is the Compute summary printed on the screen? Do you see the total time, time for load, time for compute?

```r
* Computation ended successfully.
*   Number of chunks: 4
*   Max. number of concurrent chunks (jobs): 1
*   Requested cores per job: NA
*   Load threads per chunk: 2
*   Compute threads per chunk: 4
*   Total time (s): 331.454513549805
*     Chunking setup: 0.00129866600036621
*     Data upload to cluster: 0
*     All chunks: 328.215093135834
*     Transfer results from cluster: 0
*     Merge: 3.23812174797058
*     Each chunk: 
*       queue: 
*         mean: 0
*         min: 0
*         max: 0
*       job setup: 
*         mean: 0
*         min: 0
*         max: 0
*       load: 
*         mean: 6.95802348852158
*         min: 6.30401253700256
*         max: 8.70350694656372
*       compute: 
*         mean: 75.0939234495163
*         min: 74.6092278957367
*         max: 75.3746023178101
```

2. Check the list structure using function `str()`. What are the items in the list?
```r
str(res)
List of 4
 $ rpss      : num [1, 1, 1, 1:181, 1:360] 0.1014 -0.0406 0.1024 0.2097 0.1854 ...
 $ rpss_sign : logi [1, 1, 1, 1:181, 1:360] FALSE FALSE FALSE FALSE FALSE FALSE ...
 $ rmsss     : num [1, 1, 1, 1:181, 1:360] 0.994 0.993 0.993 0.993 0.993 ...
 $ rmsss_pval: num [1, 1, 1, 1:181, 1:360] 0 0 0 0 0 0 0 0 0 0 ...
  ..
```

3. What are the dimensions of each item?
```r
dim(res$rpss)
      dat       var      time  latitude longitude 
        1         1         1       181       360 
dim(res$rpss_sign)
      dat       var      time  latitude longitude
        1         1         1       181       360
dim(res$rmsss)
      dat       var      time  latitude longitude
        1         1         1       181       360
dim(res$rmsss_pval)
      dat       var      time  latitude longitude 
        1         1         1       181       360 
```

4. Plot the maps  

Use `s2dv::PlotEquiMap` or other visualization tools to plot the maps. Check the usage of PlotEquiMap() on s2dv GitLab or type `?s2dv::PlotEquiMap`.

```r
  # Get latitude and longitude from the attributes
  lat <- as.vector(attr(exp, 'Variables')$common$latitude)
  lon <- as.vector(attr(exp, 'Variables')$common$longitude)

  # Set the color bars: check data range to decide the appropriate color bar 
  print(range(res$rpss, na.rm = TRUE))
#[1] -1.0729143  0.9982857
  brks_rpss <- seq(-1, 0.9, by = 0.1)
  print(range(res$rmsss, na.rm = TRUE))
#[1] 0.9513331 0.9997396
  brks_rmsss <- seq(0.95, 0.99, by = 0.01)

  # Plot RPSS
  ##NOTE: Remove 'fileout' to view the figure in pop-up window
  PlotEquiMap(res$rpss[1, 1, 1, , ], lon = lon, lat = lat,
              dots = res$rpss_sign[1, 1, 1, , ], dot_size = 1.5, dot_symbol = 3,
              brks = brks_rpss,
              filled.continents = FALSE, triangle_ends = c(TRUE, TRUE),
              toptitle = 'ECMWF system5 monthly mean tas RPSS 1991-2011', title_scale = 0.6,
              fileout = '~/handson_4_fig_rpss.png')

  # Plot RMSSS
  PlotEquiMap(res$rmsss[1, 1, 1, , ], lon = lon, lat = lat,
              color_fun = clim.palette('yellowred'), 
              brks = brks_rmsss,
              filled.continents = FALSE, triangle_ends = c(TRUE, TRUE),
              toptitle = 'ECMWF system5 monthly mean tas RMSSS 1991-2011', title_scale = 0.6,
              fileout = '~/handson_4_fig_rmsss.png')
```

![RPSS map](./Figures/handson_4_fig_rpss.png)
 
![RMSSS map](./Figures/handson_4_fig_rmsss.png)



**BONUS**: Try to use only one thread for loading and computing (i.e., `threads_load = 1`, `threads_compute = 1`.) Is it slower than using more threads?
It is slower with only one thread. For example, the total running time is 1214 seconds with one thread (see below)
but 331 seconds with multiple threads (see above 4.1) The loading and computing time are both more with one thread.
But the efficiency also depends on the situation of the machine. If the machine is very loaded, it can be slow for multiple threads as well.

```r
* Computation ended successfully.
*   Number of chunks: 4
*   Max. number of concurrent chunks (jobs): 1
*   Requested cores per job: NA
*   Load threads per chunk: 1
*   Compute threads per chunk: 1
*   Total time (s): 1214.47402715683
*     Chunking setup: 0.000868797302246094
*     Data upload to cluster: 0
*     All chunks: 1211.30233049393
*     Transfer results from cluster: 0
*     Merge: 3.17082786560059
*     Each chunk: 
*       queue: 
*         mean: 0
*         min: 0
*         max: 0
*       job setup: 
*         mean: 0
*         min: 0
*         max: 0
*       load: 
*         mean: 13.2360381484032
*         min: 12.5131008625031
*         max: 14.2744646072388
*       compute: 
*         mean: 289.587656855583
*         min: 286.497741699219
*         max: 294.805396080017
```

