# Hands-on Part 1
## Goal 
Know how to use different parameters of Start() to generate the desired output array.

## 1. Common-used parameters
There are several parameters of Start() usually used in every script. They help you
set up common rules and avoid mistakes.

- **retrieve**  
*Document definition: A logical value indicating whether to retrieve the data
 defined in the Start() call or to explore only its dimension lengths 
 and names, and the values for the file and inner dimensions. The default
 value is FALSE.*

If you want to retrieve data to your workstation only rather than using the whole 
startR workflow, set `retrieve = TRUE`. However, it is always recommended to use
`retrieve = FALSE` first to check the data size, so you can avoid crashing the 
workstation accidentally.

- **synonims**  
*Document definition: A named list where the names are the requested variable or 
dimension names, and the values are vectors of character strings with 
alternative names to seek for such dimension or variable.\cr\cr
In some requests, data from different sources may follow different naming 
conventions for the dimensions or variables, or even files in the same source
could have varying names. This parameter is in order for Start() to 
properly identify the dimensions or variables with different names.*

The most common synonims are lon/longitude and lat/latitude. You can also use this
parameter to rename the dimension. For example, if the variable in netCDF file has 
the time dimension named 'time' while you want to have 'ftime' dimension instead,
you can use 'ftime' as inner dimension and specify `synonims = list(ftime = c('time'))`.


- **return_vars**  
*Document definition: A named list where the names are the names of the 
variables to be fetched in the files, and the values are vectors of 
character strings with the names of the file dimension which to retrieve each
variable for, or NULL if the variable has to be retrieved only once 
from any (the first) of the involved files.*

The commonly used ones are `time = 'sdate'`, `longitude = NULL`, and `latitude = NULL`.
See FAQ [How-to-16](https://earth.bsc.es/gitlab/es/startR/-/blob/master/inst/doc/faq.md#16-use-parameter-return_vars-in-start) for more details.


```r
library(startR)

  repos <- "/esarchive/exp/ecmwf/system5_m1/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
  var <- 'tas'
  sdate <- c('20170101', '20170201')

  data <- Start(dat = repos,
                var = var,
                sdate = sdate,
                ensemble = indices(1:50),
                time = 'all',
                lat = 'all', 
                lon = 'all',
                synonims = list(lat = c('lat', 'latitude'),
                                lon = c('lon', 'longitude')),
                return_vars = list(time = 'sdate',
                                   longitude = NULL, 
                                   latitude = NULL), 
                retrieve = FALSE
               )
```

1. Run the above script. What is the data size? Check the dimension of the returned object using `attr(data, 'Dimension')`.  
```r
attr(data, 'Dimension')
     dat      var    sdate ensemble     time      lat      lon 
       1        1        2       50        7      640     1296 
```

2. Comment out the parameter 'synonims'. Run the script and see what will happen. 
Check the original dimension name of 'tas' in netCDF file by `ncdump -h /esarchive/exp/ecmwf/system5_m1/monthly_mean/tas_f6h/tas_20170101.nc |less`.  
```r
Error in Start(dat = repos, var = var, sdate = sdate, ensemble = indices(1:50),  : 
  Could not find the dimension 'lat' in the file. Either change the dimension name in your request, adjust the parameter 'dim_names_in_files' or fix the dimension name in the file.
/esarchive/exp/ecmwf/system5_m1/monthly_mean/tas_f6h/tas_20170101.nc
```

3. Recover the script. How to rename the inner dimension 'ensemble' to 'ens'?  
```r
  data <- Start(dat = repos,
                var = var,
                sdate = sdate,
                ens = indices(1:50),
                time = 'all',
                lat = 'all',
                lon = 'all',
                synonims = list(lat = c('lat', 'latitude'),
                                lon = c('lon', 'longitude'),
                                ens = c('ens', 'ensemble')),
                return_vars = list(time = 'sdate',
                                   longitude = NULL,
                                   latitude = NULL),
                retrieve = FALSE
               )
```

4. Use `str(data)` to check the data structure. Where can you find the information
of 'time', 'longitude', and 'latitude'? Can you retrieve the 'time' array?  
```r
time <- attr(data, 'Variables')$common$time
longitude <- attr(data, 'Variables')$common$longitude
latitude <- attr(data, 'Variables')$common$latitude
```

5. Comment out the parameter 'return_vars'. Can you still find the information of 'time', 'longitude', and 'latitude'?  
No. There is no information under $common nor $dat1.

6. Change 'return_vars' to `return_vars = list(time = 'sdate', longitude = 'dat', latitude = 'dat')` and check the data structure again. What is different from the original script? When will this change be beneficial?  
The longitude and latitude attributes go to $dat1. The values are the same.
```r
longitude <- attr(data, 'Variables')$dat1$longitude
latitude <- attr(data, 'Variables')$dat1$latitude
```
It does not have advantage compared to the original script (i.e., equal to NULL) because here we only have one 'dat'.

7. Following the previous question, try the script below and check again. Does it make more sense to use `return_vars = list(longitude = 'dat', latitude = 'dat')` here?   
```r
  repos <- "/esarchive/exp/ecmwf/system5_m1/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
  repos2 <- "/esarchive/exp/ecmwf/system4_m1/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
  var <- 'tas'
  sdate <- c('20170101', '20170201')

  data <- Start(dat = list(list(name = 'system5_m1', path = repos),
                           list(name = 'system4_m1', path = repos2)),
                var = var,
                sdate = sdate,
                ensemble = indices(1:50),
                time = 'all',
                lat = 'all',      
                lon = 'all',
                synonims = list(lat = c('lat', 'latitude'),
                                lon = c('lon', 'longitude')),
                return_vars = list(time = 'sdate',
                                   longitude = 'dat',      
                                   latitude = 'dat'),
                retrieve = FALSE
               )
```
Using 'dat' makes a difference now because we have two 'dat' here and they have different longitude and latitude. You can find the respective longitude/latitude under $system5_m1 and $system4_m1. 
If we still use 'NULL', the longitude/latitude will be listed under $common, and only the values of the first 'dat' (i.e., system5_m1) will be recorded.
```r
# system5_m1
longitude_system5 <- attr(data, 'Variables')$system5_m1$longitude
latitude_system5 <- attr(data, 'Variables')$system5_m1$latitude
length(longitude_system5)
[1] 1296
length(latitude_system5)
[1] 640

# system4_m1
longitude_system4 <- attr(data, 'Variables')$system4_m1$longitude
latitude_system4 <- attr(data, 'Variables')$system4_m1$latitude
length(longitude_system4)
[1] 512
length(latitude_system4)
[1] 256

```

## 2. transformation/interpolation parameters 
You can easily interpolate the data within Start(). The default transformation function
is startR::CDORemapper, a wrapper function of s2dverification::CDORemap that uses cdo inside.

The current startR version only allows the transformation with the longitude and latitude
assigned by values(). The other two ways, 'all' and indices(), are not supported now.

We are going to learn how to:  
(1) assign longitude and latitude by values()   
(2) use parameter '<dimname>_reorder', which is highly recommended to use along with (1)   
(3) use transform-related parameters in the following practice.

```r
  repos <- "/esarchive/exp/ecmwf/system5_m1/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
  var <- 'tas'
  sdate <- c('20170101')
  lon.min <- 0
  lon.max <- 359.9
  lat.min <- -90
  lat.max <- 90

  data <- Start(dat = repos,
                var = var,
                sdate = sdate,
                ensemble = indices(1:50),
                time = 'all',
                latitude = values(list(lat.min, lat.max)),
                latitude_reorder = Sort(),
                longitude = values(list(lon.min, lon.max)),
                longitude_reorder = CircularSort(0, 360),
                synonims = list(latitude = c('lat', 'latitude'),
                             longitude = c('lon', 'longitude')),
                return_vars = list(time = 'sdate',
                                   longitude = 'dat', 
                                   latitude = 'dat'),
                retrieve = FALSE
               )
```

1. Run the above script. Check the dimensions, the warning messages, and the values of 
longitude and latitude. What is the range of longitude and latitude?
```r
attr(data, 'Dimensions')
      dat       var     sdate  ensemble      time  latitude longitude 
        1         1         1        50         7       640      1296 

longitude <- attr(data, 'Variables')$dat1$longitude
range(longitude)
[1]   0.0000 359.7222

latitude <- attr(data, 'Variables')$dat1$latitude
latitude[1]
[1] -89.78488
latitude[640]
[1] 89.78488
```

2. Why 'lon.max' is 359.9 but not 360? What will happen if it is 360?
If it is 360, you only get one point of longitude. Because of `longitude_reorder = CircularSort(0, 360)`, Start() regards 0 and 360 as the same point. Therefore, we need to set a number slightly smaller than 360 but bigger than the maximum value in the original data, so we can get the whole range.

3. Now, change
	-  `latitude_reorder = Sort()` to `latitude_reorder = Sort(decreasing = TRUE)`
	-  `longitude_reorder = CircularSort(0, 360)` to `longitude_reorder = CircularSort(-180, 180)`
	-  Set `lon.min <- -180` and `lon.max <- 179.9`  
Check the values of longitude and latitude again. Is it different from the original script?
```r
attr(data, 'Dimensions')
      dat       var     sdate  ensemble      time  latitude longitude
        1         1         1        50         7       640      1296

longitude <- attr(data, 'Variables')$dat1$longitude
range(longitude)
[1] -180.0000  179.7222

latitude <- attr(data, 'Variables')$dat1$latitude
latitude[1]
[1] 89.78488
latitude[640]
[1] -89.78488

```
The dimensions are the same. The longitude range changes to [-180, 180], and the latitude sorts from 90 to -90.


Now, let us add in the transformation parameters.
  
```r
  repos <- "/esarchive/exp/ecmwf/system5_m1/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
  var <- 'tas'
  sdate <- c('20170101')
  lon.min <- 0
  lon.max <- 359.9
  lat.min <- -90
  lat.max <- 90

  data <- Start(dat = repos,
                var = var,
                sdate = sdate,
                ensemble = indices(1:50),
                time = 'all',
                latitude = values(list(lat.min, lat.max)),
                latitude_reorder = Sort(),
                longitude = values(list(lon.min, lon.max)),
                longitude_reorder = CircularSort(0, 360),
              ## transformation
                transform = CDORemapper,
                transform_extra_cells = 2,
                transform_params = list(grid = 'r360x181',
                                        method = 'conservative',
                                        crop = c(lon.min, lon.max, 
                                                 lat.min, lat.max)),
                transform_vars = c('latitude', 'longitude'),
                apply_indices_after_transform = FALSE,
              ##
                synonims = list(latitude = c('lat', 'latitude'),
                             longitude = c('lon', 'longitude')),
                return_vars = list(time = 'sdate',
                                   longitude = NULL,
                                   latitude = NULL),
                retrieve = FALSE
               )
```

4. Run the above script. Check the dimensions and the values of longitude and latitude.
```r
attr(data, 'Dimensions')
      dat       var     sdate  ensemble      time  latitude longitude 
        1         1         1        50         7       181       360 

longitude <- attr(data, 'Variables')$common$longitude
range(longitude)
[1]   0 359

latitude <- attr(data, 'Variables')$common$latitude
range(latitude)
[1] -90  90
```


## 3. Reshape parameters
Start() provides three parameters for reshaping the output dimensions: 
'merge_across_dims' and 'merge_across_dims_narm' for merging the dimensions, 
and 'split_multiselected_dims' for splitting one dimension into multiple ones.

- merge:  
The parameter 'xxx_across = yyy' indicates that the inner dimension 'xxx' is continuous along the file dimension 'yyy'. 
A common example is 'time_across = chunk', when the experiment runs through years and is saved as several chunk files. 
If the depedent relationship is indicated, you can specified the 'xxx' selector along 'yyy', not only confined to one single chunk.

Sometimes, it is more convenient to connect all 'xxx' together as one dimension instead of 'xxx' and 'yyy' two dimensions. 
To achieve this, specifying `merge_across_dims = TRUE` to merge all 'xxx' together as one dimension, and 'yyy' this dimension will disappear.

One thing to keep in mind is that the length of chosen 'xxx' at each 'yyy' should be the same, or Start() will fill NAs into the shorter 'xxx'. So, even the 'xxx' is continuous, there will be some NAs in the final 'xxx' dimension.
If the NAs are not wanted, you can use 'merge_across_dims_narm = TRUE' to remove them. 

- split:  
`split_multiselected_dims = TRUE` can split one dimension into n dimensions if this dimension is specified by an n-dimensions array.
It is useful when using experimental data attribute as the selector for the corresponding observational data, or vice versa.

```r
# An El Niño event (Apr 1957 - Mar 1959)

repos <- paste0('/esarchive/exp/ecearth/a1tr/cmorfiles/CMIP/EC-Earth-Consortium/EC-Earth3/',
                'historical/r24i1p1f1/Omon/$var$/gr/v20190312/',
                '$var$_Omon_EC-Earth3_historical_r24i1p1f1_gr_$chunk$.nc')

data <- Start(dat = repos,
              var = 'tos',
              chunk = c('195701-195712', '195801-195812', '195901-195912'),  
              time = indices(4:27), # Apr 1957 to Mar 1959
              time_across = 'chunk',
              merge_across_dims = FALSE,
              lat = values(list(-20, 20)),
              lat_reorder = Sort(),
              lon = values(list(120, 220)),
              lon_reorder = CircularSort(0, 360),
              synonims = list(lat = c('lat', 'latitude'),
                             lon = c('lon', 'longitude')),
              return_vars = list(time = 'chunk', lat = NULL, lon = NULL),
              retrieve = FALSE)
```

1. Run the above script. Check the dimensions and dimension length.
```r
attr(data, 'Dimension')
  dat   var chunk  time   lat   lon 
    1     1     3    12    58   142 
```

2. Comment out `time_across = 'chunk'`. What is the error message?
```r
Error in Start(dat = repos, var = "tos", chunk = c("195701-195712", "195801-195812",  : 
  Provided indices out of range for dimension 'time' for dataset 'dat1' (accepted range: 1 to 12).
```

3. Recover the script, and change 'merge_across_dims' to TRUE. What are the dimensions and dimension length now? Retrieve the data, and type `data[1, 1, , 10, 1]`. What is the NA pattern?
```r
attr(data, 'Dimension')
 dat  var time  lat  lon 
   1    1   36   58  142 

data[1, 1, , 10, 1]
 [1] 301.5078 300.4698 299.5681 298.6258 298.5635 299.4374 301.2602 303.2887
 [9] 304.1350       NA       NA       NA 302.9555 302.4082 302.5160 302.0977
[17] 301.0403 299.6315 299.1873 299.2484 300.3640 302.4520 304.1331 304.7454
[25] 303.7984 302.4535 301.9276       NA       NA       NA       NA       NA
[33]       NA       NA       NA       NA

```
The NAs appears at [10:12], which are the three blank months in 1957 (Jan to Mar). 
Also at [28:36], which are the nine blank months in 1959 (Apr to Dec). 
Start() put the NAs at the end of each year (chunk).

4. The time length we want is actually 24 months, from April 1957 to March 1959. To remove those additional NAs, add `merge_across_dims_narm = TRUE` in the above code.
What are the dimensions and dimension length now? Check `data[1, 1, , 10, 1]` again. Is there any NA?
```r
dim(data)
 dat  var time  lat  lon 
   1    1   24   58  142 

data[1, 1, , 10, 1]
 [1] 301.5078 300.4698 299.5681 298.6258 298.5635 299.4374 301.2602 303.2887
 [9] 304.1350 302.9555 302.4082 302.5160 302.0977 301.0403 299.6315 299.1873
[17] 299.2484 300.3640 302.4520 304.1331 304.7454 303.7984 302.4535 301.9276
```
With `merge_across_dims_narm = TRUE`, the additional NAs are removed. The time length is 24 now.


In the following script, 'merge' and 'split' parameters are used. we are going to load experimental data first, then use its attirbute to 
load the corresponding observational data.

```r
# exp
repos_exp <- paste0('/esarchive/exp/ecearth/a1tr/cmorfiles/CMIP/EC-Earth-Consortium/EC-Earth3/',
                    'historical/r24i1p1f1/Amon/$var$/gr/v20190312/',
                    '$var$_Amon_EC-Earth3_historical_r24i1p1f1_gr_$sdate$01-$sdate$12.nc')

exp <- Start(dat = repos_exp,
             var = 'tas',
             sdate = as.character(c(2005:2008)),
             time = indices(1:3),
             lat = 'all',
             lon = 'all',
             synonims = list(lat = c('lat', 'latitude'),
                             lon = c('lon', 'longitude')),
             return_vars = list(lon = NULL,
                                lat = NULL,
                                time = 'sdate'),
             retrieve = FALSE)

lats <- attr(exp, 'Variables')$common$lat
lons <- attr(exp, 'Variables')$common$lon
dates <- attr(exp, 'Variables')$common$time

# obs
repos_obs <- '/esarchive/recon/ecmwf/erainterim/monthly_mean/$var$_f6h/$var$_$date$.nc'

obs <- Start(dat = repos_obs,
             var = 'tas',
             date = unique(format(dates, '%Y%m')),
             time = values(dates),
             time_across = 'date',
             merge_across_dims = TRUE,
             split_multiselected_dims = TRUE,
             lat = values(lats),
             lon = values(lons),
             synonims = list(lat = c('lat', 'latitude'),
                             lon = c('lon', 'longitude')),
             return_vars = list(lon = NULL,
                                lat = NULL,
                                time = 'date'),
             retrieve = FALSE)

```

1. Check the dimension of 'dates', the time attribute of experimental data. How many dimensions does it have?
```r
dim(dates)
sdate  time 
    4     3 
```
It has two dimesnions.

2. We want to get the same dimension structure of observational data as the experimental one.
However, the netCDF files of the two are different. Each experimental data has 12 months while each observational data has 1 month.
To generate the same dimensions, we need to **merge 'time' across 'date' first, then split 'time' into 'sdate' and 'time'.**  
Check the two output dimensions of the above script. Are they the same?
```r
attr(exp, 'Dimensions')
  dat   var sdate  time   lat   lon 
    1     1     4     3   256   512 

attr(obs, 'Dimensions')
  dat   var sdate  time   lat   lon
    1     1     4     3   256   512

```

3. Comment out `split_multiselected_dims = TRUE` in obs. What is the dimension now?
```r
attr(obs, 'Dimensions')
 dat  var time  lat  lon 
   1    1   12  256  512 
```

4. What will happen if `split_multiselected_dims = TRUE` but `merge_across_dims = FALSE`? Does it make sense?
```r
attr(obs, 'Dimensions')
  dat   var  date sdate  time   lat   lon 
    1     1    12     4     3   256   512 
```
It does not make sense to have 'sdate' and 'time' dimensions and 'date' dimension together. They are repetitive. Always merge the dimensions before splitting!

## 4. Others
The parameters we did not mention in this hands-on (but you might use): 'pattern_dims', 'metadata_dims', 'path_glob_permissive'. You can check the startR document or go to FAQs and usecase on startR GitLab to learn more.
