# Hands-on Part 2
## Goal
Learn how to use the whole startR workflow to calculate skill score.
In this case, the ensemble-adjusted Continuous Ranked Probability Score (CRPS) and the root mean square error skill score (RMSSS) are used.
To make the process faster, the required data size is small here, and we can run the execution on workstation.

## 1. Define data
We want to verify the experimental data with the observational data by the skill scores. 
The first step is use Start() to define the data.

Hints:  
(1) The experimental data files are under `/esarchive/exp/ecmwf/system4_m1/monthly_mean/`, and the observational data files are under `/esarchive/recon/ecmwf/erainterim/monthly_mean/`.  
(2) The variable we want to use is 'tas' under 'tas_f6h/'.  
(3) The required time (sdate) period is November 1981 to 2016.  
(4) Read global data.  
(5) Take only the first time step.  
(6) To make the data smaller, we only take the first two ensemble members from the experimental data.  
(7) Because we are going to use the whole startR workflow, we only need to create a pointer to the data repository rather than retrieve it to the workstation.

```r
library(startR)

# exp data
  repos_exp <- paste0('/esarchive/exp/ecmwf/system4_m1/monthly_mean/',
                      '$var$_f6h/$var$_$sdate$.nc')
  sdates_exp <- sapply(1981:2016, function(x) paste0(x, '1101'))

  exp <- Start(dat = repos_exp,
               var = 'tas',
               sdate = sdates_exp,
               time = indices(1),
               ensemble = indices(1:2),
               latitude = 'all',
               longitude = 'all',
               synonims = list(longitude = c('lon', 'longitude'),
                               latitude = c('lat', 'latitude')),
               retrieve = F)

# obs data
  repos_obs <- paste0('/esarchive/recon/ecmwf/erainterim/monthly_mean/',
                      '$var$_f6h/$var$_$sdate$.nc')
  sdates_obs <- sapply(1981:2016, function(x) paste0(x, '11'))

  obs <- Start(dat = repos_obs,
                     var = 'tas',
                     sdate = sdates_obs,
                     time = indices(1),
                     latitude = 'all',
                     longitude = 'all',
                     synonims = list(longitude = c('lon', 'longitude'),
                                     latitude = c('lat', 'latitude')),
                     retrieve = F)
```

Question:  

1. What are the dimensions of these two data?
```r
attr(exp, 'Dimensions')
      dat       var     sdate      time  ensemble  latitude longitude 
        1         1        36         1         2       256       512 

attr(obs, 'Dimensions')
      dat       var     sdate      time  latitude longitude 
        1         1        36         1       256       512 
```

2. What is the size of these two data?
exp: 72Mb; obs: 36Mb


## 2. Define operation and workflow
It is recommended to define the function and write Step() together, because the latter one helps you clarify the input and output dimensions of the function.

In the self-defined function, we want to use the two functions, 'SpecsVerification::EnsCrps' and 's2dv:::.RMSSS', to calculate the skill score.
You can check the first function by typing `?SpecsVerification::EnsCrps`, and the second function on [s2dv GitLab](https://earth.bsc.es/gitlab/es/s2dv/-/blob/master/R/RMSSS.R).  

Hint:  
(1) The self-defined function is for exp and obs data. Therefore, target_dims and output_dims in Step() should be a list with two vector elements.
(2) Check the functions 'EnsCrps' and '.RMSSS'. What are the minimum required dimensions of inputs of each function? These dimensions should be the target_dims in Step().  
(3) To return the two skill scores together, put them in a list with two elements.  
(4) What are the dimensions of outputs of each function? These dimension should be the output_dims in Step(). 
(5) The first input of AddStep() should also be a list containing exp and obs.

```r
  # self-defined function
  func <- function(x, y) {
    # CRPS
    crps <- mean(SpecsVerification::EnsCrps(x, y, R.new = Inf))

    # RMSSS
    # exp: [sdate, member_exp]
    # obs: [sdate, member_obs]
    # obs only has [sdate] now. Add one more dim for it.
    y <- s2dv::InsertDim(y, posdim = 2, lendim = 1, name = 'ensemble')
    rmsss <- mean(s2dv:::.RMSSS(x, y, time_dim = 'sdate', memb_dim = 'ensemble', pval = FALSE)$rmsss)

    return(list(crps = crps, rmsss = rmsss))
  }
  step <- Step(func, target_dims = list(c('sdate', 'ensemble'), c('sdate')),
                     output_dims = list(crps = NULL, rmsss = NULL))
  wf <- AddStep(list(exp, obs), step)
```

Question:  
1. Which dimensions are used in operation? Which dimensions are free?
'sdate' and 'ensemble' are used in operation. The other dimensions (dat, var, time, latitude, and longitude) are not necessary for the function. These free dimensions will be subsetted by multiApply during the execution.

2. Can we use more dimensions as target_dims? What are the disadvantages?  
It still works if putting more dimensions as target_dims, but we will lose the choices for chunking in the next step. Also, the lighter the function is, the quicker the operation runs.  



## 3. Execute locally
To avoid potential technical problems in the connection and configuration, we choose to run the execution locally. Please remember that it is recommended to submit the job to HPCs if the operation is heavy.

Hint:  
(1) Use the free dimensions to chunk.  
(2) It is safe to divide the data into pieces of which the size each is 1/2-1/3 of RAM. In this case, the data size is only around 100Mb, so you can play with it without worrying to crash your workstation.

```r
  res <- Compute(wf$crps,
                 chunks = list(latitude = 2,
                               longitude = 2))
```

## 4. Check the results

1. Check the list structure. What is the length of the list?
```r
 str(res)
List of 2
 $ crps : num [1, 1, 1, 1:256, 1:512] 1.02 1.06 1.09 1.12 1.1 ...
 $ rmsss: num [1, 1, 1, 1:256, 1:512] 0.991 0.991 0.99 0.989 0.989 ...
```

2. What does the dimension look like?
```r
dim(res$crps)
      dat       var      time  latitude longitude 
        1         1         1       256       512 
dim(res$rmsss)
      dat       var      time  latitude longitude
        1         1         1       256       512
```

3. Plot the map  
Use `s2dv::PlotEquiMap` or other tools to plot the map.
```r
  library(s2dv)
  # manually create the lon and lat vector
  lon <- seq(from = 0, to = 359, length.out = 512)
  lat <- seq(from = 90, to = -90, length.out = 256)

  # Set the color bar 
  brks_crps <- seq(min(res$crps, na.rm = TRUE), max(res$crps, na.rm = TRUE), length.out = 11)
  brks_rmsss <- seq(min(res$rmsss, na.rm = TRUE), max(res$rmsss, na.rm = TRUE), length.out = 11)

  # Plot crps
  PlotEquiMap(res$crps[1, 1, 1, , ], lon, lat,
              color_fun = clim.palette('yellowred'), brks = brks_crps,
              filled.continents = FALSE, triangle_ends = c(TRUE, TRUE),
              toptitle = 'ECMWF monthly mean tas CRPS 2012-2016', title_scale = 0.6)

  # Plot rmsss
  PlotEquiMap(res$rmsss[1, 1, 1, , ], lon, lat,
              color_fun = clim.palette('yellowred'), brks = brks_rmsss,
              filled.continents = FALSE, triangle_ends = c(TRUE, TRUE),
              toptitle = 'ECMWF monthly mean tas RMSSS 2012-2016', title_scale = 0.6)

```

4. (bonus) We create lat and lon vectors by ourselves above. However, the actual values are slightly different.
Can you get the lon and lat vectors from the data attributes?

```r
  repos_exp <- paste0('/esarchive/exp/ecmwf/system4_m1/monthly_mean/',
                      '$var$_f6h/$var$_$sdate$.nc')
  sdates_exp <- sapply(2012:2016, function(x) paste0(x, sprintf('%02d', 1:12), '01'))

  exp <- Start(dat = repos_exp,
               var = 'tas',
               sdate = sdates_exp,
               time = indices(1),
               ensemble = indices(1:2),
               latitude = 'all',
               longitude = 'all',
               synonims = list(longitude = c('lon', 'longitude'),
                               latitude = c('lat', 'latitude')),
               return_vars = list(latitude = NULL, longitude = NULL),
               retrieve = F)

  lon <- as.vector(attr(exp, 'Variables')$common$longitude)
  lat <- as.vector(attr(exp, 'Variables')$common$latitude)

```

