# Hands-on 1: Seasonal forecast verification

## Goal
Learn how to use the startR workflow to finish a piece of analysis, including defining and pre-processing the desired data,
defining the operation, building the workflow, and executing the operation. 

In this use case, the ensemble-adjusted Continuous Ranked Probability Score (CRPS) and the root mean square error skill score (RMSSS) are computed to verify the forecast.
To make the process faster, the required data size is small here so we can run the execution on workstation.

## 1. Define data
The first step of the analysis is use Start() to identify the data needed.

Hints:  
(1) The experimental data files are under `/esarchive/exp/ecmwf/system4_m1/monthly_mean/`, and the observational data files are under `/esarchive/recon/ecmwf/erainterim/monthly_mean/`.  
(2) The variable we want to use is 'tas'.  
(3) The required time (sdate) period is November 1981 to 2016.  
(4) Read global data (i.e., full spatial grids.)  
(5) Take only the first time step.  
(6) To make the data smaller, we only take the first two ensemble members from the experimental data.  
(7) Because we are going to use the whole startR workflow, we only need to create a pointer to the data repository rather than retrieve it to the workstation.

```r
library(startR)

# exp data
  repos_exp <- paste0('/esarchive/exp/ecmwf/system4_m1/monthly_mean/',
                      '$var$_f6h/$var$_$sdate$.nc')
  sdates_exp <- sapply(1981:2016, function(x) paste0(x, '1101'))

  exp <- Start(dat = repos_exp,
               var = ___<fill>___,
               sdate = sdates_exp,
               time = ___<fill>___,
               ensemble = ___<fill>___,
               latitude = ___<fill>___,
               longitude = ___<fill>___,
               synonims = list(longitude = c('lon', 'longitude'),
                               latitude = c('lat', 'latitude')),
               return_vars = list(latitude = NULL, longitude = NULL, time = NULL),
               retrieve = ___<fill>___)

# obs data
  repos_obs <- paste0('/esarchive/recon/ecmwf/erainterim/monthly_mean/',
                      '$var$_f6h/$var$_$sdate$.nc')
  sdates_obs <- sapply(1981:2016, function(x) paste0(x, '11'))

  obs <- Start(dat = repos_obs,
               var = ___<fill>___,
               sdate = sdates_obs,
               time = ___<fill>___,
               latitude = ___<fill>___,
               longitude = ___<fill>___,
               synonims = list(longitude = c('lon', 'longitude'),
                               latitude = c('lat', 'latitude')),
               return_vars = list(latitude = NULL, longitude = NULL, time = NULL),
               retrieve = ___<fill>___)
```

Question:  

1. What are the dimensions of these two data?

2. What is the size of these two data?


## 2. Define operation and workflow
It is recommended to define the function and write Step() together, because the latter one helps you clarify the input and output dimensions of the function.

In the self-defined function, we want to use the two functions, 'SpecsVerification::EnsCrps' and 's2dv:::.RMSSS', to calculate the skill score.
You can check the first function by typing `?SpecsVerification::EnsCrps`, and the second function on [s2dv GitLab](https://earth.bsc.es/gitlab/es/s2dv/-/blob/master/R/RMSSS.R).  

Hint:  
(1) The self-defined function is for exp and obs data. Therefore, target_dims and output_dims in Step() should be a list with two vector elements.
(2) Check the functions 'EnsCrps' and '.RMSSS'. What are the minimum required dimensions of inputs of each function? These dimensions should be the 'target_dims' in Step().  
(3) To return the two skill scores ('crps' and 'rmsss') together, put them in a list with two elements.  
(4) What are the dimensions of outputs? These dimensions should be the 'output_dims' in Step(). 
(5) The first input of AddStep() should also be a list containing exp and obs.

```r
  # self-defined function
  func <- function(exp, obs) {
    # CRPS
    crps <- mean(SpecsVerification::EnsCrps(exp, obs, R.new = Inf), na.rm = TRUE)

    # RMSSS
    # obs only has [sdate] now. Add one more dim for it.
    obs <- s2dv::InsertDim(obs, posdim = 2, lendim = 1, name = 'ensemble')
    rmsss <- mean(s2dv:::.RMSSS(exp, obs, time_dim = 'sdate', dat_dim = 'ensemble', pval = FALSE)$rmsss)

    return(___<fill>___)
  }
  step <- Step(func, target_dims = ___<fill>___,
                     output_dims = ___<fill>___)
  wf <- AddStep(___<fill>___, step)
```

Question:  
1. Which dimensions are used in operation? Which dimensions are free?

2. Can we use more dimensions as target_dims? What are the disadvantages?  


## 3. Execute locally
To avoid potential technical problems in the connection and configuration, 
we choose to run the execution locally. 
Noted that it is recommended to submit jobs to HPCs if the data size is big or 
the operation is heavy.

Hint:  
(1) Use the free dimensions (i.e., those are not 'target_dims' in Step()) to chunk.  
(2) It is safe to divide the data into pieces of which the size each is 1/2-1/3 of RAM. In this case, the data size is only around 100Mb, so you can play with it without 
worrying about crashing the workstation.  
(3) You will see some error messages like "Error in R_nc4_open: No such file or directory",
which are ignorable.

```r
  res <- Compute(wf$crps,
                 chunks = list(___<fill>___))
```

## 4. Check the results

1. Check the list structure using function `str()`. What are the items in the list?

2. What do the dimensions of each item look like?

3. Plot the map  
Use `s2dv::PlotEquiMap` or other visualization tools to plot the map.
```r
  # Get latitude and longitude from the attributes
  lat <- as.vector(attr(exp, 'Variables')$common$latitude)
  lon <- as.vector(attr(exp, 'Variables')$common$longitude)

  # Set the color bar 
  brks_crps <- seq(min(res$crps, na.rm = TRUE), max(res$crps, na.rm = TRUE), length.out = 11)
  brks_rmsss <- seq(min(res$rmsss, na.rm = TRUE), max(res$rmsss, na.rm = TRUE), length.out = 11)

  library(s2dv)
  # Plot crps
  PlotEquiMap(___<fill>___, lon, lat,
              color_fun = clim.palette('yellowred'), brks = brks_crps,
              filled.continents = FALSE, triangle_ends = c(TRUE, TRUE),
              toptitle = 'ECMWF monthly mean tas CRPS 2012-2016', title_scale = 0.6, 
              fileout = '~/handson_1_fig_crps.png')

  # Plot rmsss
  PlotEquiMap(___<fill>___, lon, lat,
              color_fun = clim.palette('yellowred'), brks = brks_rmsss,
              filled.continents = FALSE, triangle_ends = c(TRUE, TRUE),
              toptitle = 'ECMWF monthly mean tas RMSSS 2012-2016', title_scale = 0.6,
              fileout = '~/handson_1_fig_rmsss.png')

```
