# Hands-on 1: Load data by startR

## Goal
Use startR to load the data and learn how to adjust data structure while loading data.

## 0. Load required packages

```r
# Clean the session
rm(list = ls())

library(startR)
```
**Data description**:  
We will use two datasets in the hands-on. The experiment data are Meteo-France System 7 from ECMWF, and the observation ones are ERA5 from ECMWF. The data have been first processed into monthly mean data and stored in our data archive (esarchive).

We're going to analyze the near-surface temperature (short name: tas) for seasonal forecast. We will focus on the Europe region (roughly 20W-40E, 20N-80N). The hindcast years are 1993 to 2016, and the forecast year is 2020. The initial month is November. To speed up the practice, we will only load the first two forecast time steps, but all the ensemble members are used to give a less biased result.

## 1. Load experimental data from data repository

### 1.1 Hindcast data

Understand the following script, run it, and check the result. 

```r
  # Use this one if on workstation or nord3 (have access to /esarchive)
  path_exp <- "/esarchive/exp/meteofrance/system7c3s/monthly_mean/$var$_f6h/$var$_$syear$.nc"
  #----------------------------------------------------------------------
  # Run these two lines if you're on Marenostrum4 and log in with training account
  prefix <- '/gpfs/scratch/bsc32/bsc32734/bsc_training_2023/R_handson/'
  path_exp <- paste0(prefix, path_exp)
  #----------------------------------------------------------------------

  var <- 'tas'
  sdate_hcst <- paste0(1993:2016, '1101')
  lon.min <- -20
  lon.max <- 40
  lat.min <- 20
  lat.max <- 80
```

Use Start() to load the data.

```r
  hcst <- Start(dat = path_exp,
                var = var,
                syear = sdate_hcst,
                ensemble = 'all',
                time = 1:2,
                latitude = values(list(lat.min, lat.max)),
                latitude_reorder = Sort(),
                longitude = values(list(lon.min, lon.max)),
                longitude_reorder = CircularSort(-180, 180),
                synonims = list(syear = c('syear', 'sdate'),
                                latitude = c('lat', 'latitude'),
                                longitude = c('lon', 'longitude')),
                return_vars = list(time = 'syear',
                                   longitude = NULL, latitude = NULL),
                retrieve = TRUE)
```
   
### 1.2 Forecast data

The forecast data are from the same dataset as hindcast, but with different years. 
Therefore, they share the same data path and strucutre. 
Try to take the Start() call above and modify it to load the forecast data (hint: the start year is 2020.)

```r
  sdate_fcst <- '20201101'

  fcst <- Start(dat = path_exp,
                var = var,
                syear = sdate_fcst,
                ensemble = 'all',
                time = 1:2, 
                latitude = values(list(lat.min, lat.max)),
                latitude_reorder = Sort(),
                longitude = values(list(lon.min, lon.max)),
                longitude_reorder = CircularSort(-180, 180),
                synonims = list(syear = c('syear', 'sdate'),
                                latitude = c('lat', 'latitude'),
                                longitude = c('lon', 'longitude')),
                return_vars = list(time = 'syear',
                                   longitude = NULL, latitude = NULL),
                retrieve = TRUE)
```

### 1.3 Observational data

We need the corresponding observational data to compare with the experimental data.
So, the observational data should be loaded as the same dimensions as the experimental ones.
To achieve this, we can use the metadata of the experimental data as the selectors for observational data. But be careful with the usage! We must verify the correctness and applicability first. 

**Get the time values from hindcast data**
Check the time attributes of `hcst`: Is it correct? 

```r
dim(attributes(hcst)$Variables$common$time)
#syear  time
#   24     2

str(attributes(hcst)$Variables$common$time)
# POSIXct[1:48], format: "1993-12-01" "1994-12-01" "1995-12-01" "1996-12-01" "1997-12-01" ...
```

The values are not correct since they should start from November, not December. 
But the array has the correct dimensions and we can take advantage of it. 
What we're going to do here is to tune the values one month ahead so we can have the correct dates. 
(ps., `lubridate` is a useful R package for time value manipulation!)

```r
attributes(hcst)$Variables$common$time <- attributes(hcst)$Variables$common$time - lubridate::days(1)
date_string <- format(attributes(hcst)$Variables$common$time, '%Y%m')
sdate_obs <- array(date_string, dim = c(syear = 24, time = 2))
```

```r
  path_obs <- '/esarchive/recon/ecmwf/era5/monthly_mean/$var$_f1h-r1440x721cds/$var$_$syear$.nc'
  #----------------------------------------------------------------------
  # Run these two lines if you're on Marenostrum4 and log in with training account
  prefix <- '/gpfs/scratch/bsc32/bsc32734/bsc_training_2023/R_handson/'
  path_obs <- paste0(prefix, path_obs)
  #----------------------------------------------------------------------

  obs <- Start(dat = path_obs,
               var = var,
               syear = sdate_obs,
               split_multiselected_dims = TRUE,
               latitude = values(list(lat.min, lat.max)),
               latitude_reorder = Sort(),
               longitude = values(list(lon.min, lon.max)),
               longitude_reorder = CircularSort(-180, 180),
               transform = CDORemapper,
               transform_params = list(grid = '/gpfs/scratch/bsc32/bsc32734/bsc_training_2023/R_handson/griddes_system7c3s.txt',
                                       method = 'bilinear'),
               transform_vars = c('latitude', 'longitude'),
               synonims = list(syear = c('syear', 'sdate'),
                               latitude = c('lat', 'latitude'),
                               longitude = c('lon', 'longitude')),
               return_vars = list(time = 'syear',
                                  longitude = NULL, latitude = NULL),
               retrieve = TRUE)
```

## 2. Check if the datasets are consistent

```r
lat_hcst <- attributes(hcst)$Variables$common$latitude
lon_hcst <- attributes(hcst)$Variables$common$longitude
lat_obs <- attributes(obs)$Variables$common$latitude
lon_obs <- attributes(obs)$Variables$common$longitude

identical(c(lat_obs), c(lat_hcst))
[1] TRUE
identical(c(lon_obs), c(lon_hcst))
[1] TRUE

time_hcst <- attributes(hcst)$Variables$common$time
time_obs <- attributes(obs)$Variables$common$time

identical(format(time_hcst, '%Y%m'), format(time_obs, '%Y%m'))
[1] TRUE
```
