# Hands-on Part 1
## Goal 
Know how to use different parameters of Start() to generate the desired output array.

## 1. Common-used parameters
There are several parameters of Start() usually used in every script. They help you
set up common rules and avoid mistakes.

- **retrieve**  
*Document definition: A logical value indicating whether to retrieve the data
 defined in the Start() call or to explore only its dimension lengths 
 and names, and the values for the file and inner dimensions. The default
 value is FALSE.*

If you want to retrieve data to your workstation only rather than using the whole 
startR workflow, set `retrieve = TRUE`. However, it is always recommended to use
`retrieve = FALSE` first to check the data size, so you can avoid crashing the 
workstation accidentally.

- **synonims**  
*Document definition: A named list where the names are the requested variable or 
dimension names, and the values are vectors of character strings with 
alternative names to seek for such dimension or variable.\cr\cr
In some requests, data from different sources may follow different naming 
conventions for the dimensions or variables, or even files in the same source
could have varying names. This parameter is in order for Start() to 
properly identify the dimensions or variables with different names.*

The most common synonims are lon/longitude and lat/latitude. You can also use this
parameter to rename the dimension. For example, if the variable in netCDF file has 
the time dimension named 'time' while you want to have 'ftime' dimension instead,
you can use 'ftime' as inner dimension and specify `synonims = list(ftime = c('time'))`.


- **return_vars**  
*Document definition: A named list where the names are the names of the 
variables to be fetched in the files, and the values are vectors of 
character strings with the names of the file dimension which to retrieve each
variable for, or NULL if the variable has to be retrieved only once 
from any (the first) of the involved files.*

The commonly used ones are `time = 'sdate'`, `longitude = NULL`, and `latitude = NULL`.
See FAQ [How-to-16](https://earth.bsc.es/gitlab/es/startR/-/blob/master/inst/doc/faq.md#16-use-parameter-return_vars-in-start) for more details.


```r
library(startR)

  repos <- "/esarchive/exp/ecmwf/system5_m1/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
  var <- 'tas'
  sdate <- c('20170101', '20170201')

  data <- Start(dat = repos,
                var = var,
                sdate = sdate,
                ensemble = indices(1:50),
                time = 'all',
                lat = 'all', 
                lon = 'all',
                synonims = list(lat = c('lat', 'latitude'),
                                lon = c('lon', 'longitude')),
                return_vars = list(time = 'sdate',
                                   longitude = NULL, 
                                   latitude = NULL), 
                retrieve = FALSE
               )
```

1. Run the above script. What is the data size? Check the dimension of the returned object using `attr(data, 'Dimension')`. 

2. Comment out the parameter 'synonims'. Run the script and see what will happen. 
Check the original dimension name of 'tas' in netCDF file by `ncdump -h /esarchive/exp/ecmwf/system5_m1/monthly_mean/tas_f6h/tas_20170101.nc |less`.

3. Recover the script. How to rename the inner dimension 'ensemble' to 'ens'? 

4. Use `str(data)` to check the data structure. Where can you find the information
of 'time', 'longitude', and 'latitude'? Can you retrieve the 'time' array?

5. Comment out the parameter 'return_vars'. Can you still find the information of 'time', 'longitude', and 'latitude'? 

6. Change 'return_vars' to `return_vars = list(time = 'sdate', longitude = 'dat', latitude = 'dat')` and check the data structure again. What is different from the original script? When will this change be beneficial? Try the following script and check again.

```r
  repos <- "/esarchive/exp/ecmwf/system5_m1/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
  repos2 <- "/esarchive/exp/ecmwf/system4_m1/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
  var <- 'tas'
  sdate <- c('20170101', '20170201')

  data <- Start(dat = list(list(name = 'system5_m1', path = repos),
                           list(name = 'system4_m1', path = repos2)),
                var = var,
                sdate = sdate,
                ensemble = indices(1:50),
                time = 'all',
                lat = 'all',      
                lon = 'all',
                synonims = list(lat = c('lat', 'latitude'),
                                lon = c('lon', 'longitude')),
                return_vars = list(time = 'sdate',
                                   longitude = 'dat',      
                                   latitude = 'dat'),
                retrieve = FALSE
               )
```


## 2. transformation/interpolation parameters 
You can easily interpolate the data within Start(). The default transformation function
is startR::CDORemapper, a wrapper function of s2dverification::CDORemap that uses cdo inside.

The current startR version only allows the transformation with the longitude and latitude
assigned by values(). The other two ways, 'all' and indices(), are not supported now.

We are going to learn how to:  
(1) assign longitude and latitude by values()   
(2) use parameter '<dimname>_reorder', which is highly recommended to use along with (1)   
(3) use transform-related parameters in the following practice.

```r
  repos <- "/esarchive/exp/ecmwf/system5_m1/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
  var <- 'tas'
  sdate <- c('20170101')
  lon.min <- 0
  lon.max <- 359.9
  lat.min <- -90
  lat.max <- 90

  data <- Start(dat = repos,
                var = var,
                sdate = sdate,
                ensemble = indices(1:50),
                time = 'all',
                latitude = values(list(lat.min, lat.max)),
                latitude_reorder = Sort(),
                longitude = values(list(lon.min, lon.max)),
                longitude_reorder = CircularSort(0, 360),
                synonims = list(latitude = c('lat', 'latitude'),
                             longitude = c('lon', 'longitude')),
                return_vars = list(time = 'sdate',
                                   longitude = 'dat', 
                                   latitude = 'dat'),
                retrieve = FALSE
               )
```

1. Run the above script. Check the dimensions, the warning messages, and the values of 
longitude and latitude. What is the range of longitude and latitude?

2. Why 'lon.max' is 359.9 but not 360? What will happen if it is 360?

3. Now, change
        -  `latitude_reorder = Sort()` to `latitude_reorder = Sort(decreasing = TRUE)`
        -  `longitude_reorder = CircularSort(0, 360)` to `longitude_reorder = CircularSort(-180, 180)`
        -  Set `lon.min <- -180` and `lon.max <- 179.9`
Check the values of longitude and latitude again. Is it different from the original script?



Now, let us add in the transformation parameters.
  
```r
  repos <- "/esarchive/exp/ecmwf/system5_m1/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
  var <- 'tas'
  sdate <- c('20170101')
  lon.min <- 0
  lon.max <- 359.9
  lat.min <- -90
  lat.max <- 90

  data <- Start(dat = repos,
                var = var,
                sdate = sdate,
                ensemble = indices(1:50),
                time = 'all',
                latitude = values(list(lat.min, lat.max)),
                latitude_reorder = Sort(),
                longitude = values(list(lon.min, lon.max)),
                longitude_reorder = CircularSort(0, 360),
              ## transformation
                transform = CDORemapper,
                transform_extra_cells = 2,
                transform_params = list(grid = 'r360x181',
                                        method = 'conservative',
                                        crop = c(lon.min, lon.max, 
                                                 lat.min, lat.max)),
                transform_vars = c('latitude', 'longitude'),
                apply_indices_after_transform = FALSE,
              ##
                synonims = list(latitude = c('lat', 'latitude'),
                             longitude = c('lon', 'longitude')),
                return_vars = list(time = 'sdate',
                                   longitude = NULL,
                                   latitude = NULL),
                retrieve = FALSE
               )
```
4. Run the above script. Check the dimensions and the values of longitude and latitude.


## 3. Reshape parameters
Start() provides three parameters for reshaping the output dimensions: 
'merge_across_dims' and 'merge_across_dims_narm' for merging the dimensions, 
and 'split_multiselected_dims' for splitting one dimension into multiple ones.

- merge:  
The parameter 'xxx_across = yyy' indicates that the inner dimension 'xxx' is continuous along the file dimension 'yyy'. 
A common example is 'time_across = chunk', when the experiment runs through years and is saved as several chunk files. 
If the depedent relationship is indicated, you can specified the 'xxx' selector along 'yyy', not only confined to one single chunk.

Sometimes, it is more convenient to connect all 'xxx' together as one dimension instead of 'xxx' and 'yyy' two dimensions. 
To achieve this, specifying `merge_across_dims = TRUE` to merge all 'xxx' together as one dimension, and 'yyy' this dimension will disappear.

One thing to keep in mind is that the length of chosen 'xxx' at each 'yyy' should be the same, or Start() will fill NAs into the shorter 'xxx'. So, even the 'xxx' is continuous, there will be some NAs in the final 'xxx' dimension.
If the NAs are not wanted, you can use 'merge_across_dims_narm = TRUE' to remove them. 

- split:  
`split_multiselected_dims = TRUE` can split one dimension into n dimensions if this dimension is specified by an n-dimensions array.
It is useful when using experimental data attribute as the selector for the corresponding observational data, or vice versa.

```r
# An El Niño event (Apr 1957 - Mar 1959)

repos <- paste0('/esarchive/exp/ecearth/a1tr/cmorfiles/CMIP/EC-Earth-Consortium/EC-Earth3/',
                'historical/r24i1p1f1/Omon/$var$/gr/v20190312/',
                '$var$_Omon_EC-Earth3_historical_r24i1p1f1_gr_$chunk$.nc')

data <- Start(dat = repos,
              var = 'tos',
              chunk = c('195701-195712', '195801-195812', '195901-195912'),  
              time = indices(4:27), # Apr 1957 to Mar 1959
              time_across = 'chunk',
              merge_across_dims = FALSE,
              lat = values(list(-20, 20)),
              lat_reorder = Sort(),
              lon = values(list(120, 220)),
              lon_reorder = CircularSort(0, 360),
              synonims = list(lat = c('lat', 'latitude'),
                             lon = c('lon', 'longitude')),
              return_vars = list(time = 'chunk', lat = NULL, lon = NULL),
              retrieve = FALSE)
```

1. Run the above script. Check the dimensions and dimension length.

2. Comment out `time_across = 'chunk'`. What is the error message?

3. Recover the script, and change 'merge_across_dims' to TRUE. What are the dimensions and dimension length now? Retrieve the data, and type `data[1, 1, , 10, 1]`. What is the NA pattern?

4. The time length we want is actually 24 months, from April 1957 to March 1959. To remove those additional NAs, add `merge_across_dims_narm = TRUE` in the above code.
What are the dimensions and dimension length now? Check `data[1, 1, , 10, 1]` again. Is there any NA?


In the following script, 'merge' and 'split' parameters are used. we are going to load experimental data first, then use its attirbute to 
load the corresponding observational data.

```r
# exp
repos_exp <- paste0('/esarchive/exp/ecearth/a1tr/cmorfiles/CMIP/EC-Earth-Consortium/EC-Earth3/',
                    'historical/r24i1p1f1/Amon/$var$/gr/v20190312/',
                    '$var$_Amon_EC-Earth3_historical_r24i1p1f1_gr_$sdate$01-$sdate$12.nc')

exp <- Start(dat = repos_exp,
             var = 'tas',
             sdate = as.character(c(2005:2008)),
             time = indices(1:3),
             lat = 'all',
             lon = 'all',
             synonims = list(lat = c('lat', 'latitude'),
                             lon = c('lon', 'longitude')),
             return_vars = list(lon = NULL,
                                lat = NULL,
                                time = 'sdate'),
             retrieve = FALSE)

lats <- attr(exp, 'Variables')$common$lat
lons <- attr(exp, 'Variables')$common$lon
dates <- attr(exp, 'Variables')$common$time

# obs
repos_obs <- '/esarchive/recon/ecmwf/erainterim/monthly_mean/$var$_f6h/$var$_$date$.nc'

obs <- Start(dat = repos_obs,
             var = 'tas',
             date = unique(format(dates, '%Y%m')),
             time = values(dates),
             time_across = 'date',
             merge_across_dims = TRUE,
             split_multiselected_dims = TRUE,
             lat = values(lats),
             lon = values(lons),
             synonims = list(lat = c('lat', 'latitude'),
                             lon = c('lon', 'longitude')),
             return_vars = list(lon = NULL,
                                lat = NULL,
                                time = 'date'),
             retrieve = FALSE)

```

1. Check the dimension of 'dates', the time attribute of experimental data. How many dimensions does it have?

2. We want to get the same dimension structure of observational data as the experimental one.
However, the netCDF files of the two are different. Each experimental data has 12 months while each observational data has 1 month.
To generate the same dimensions, we need to **merge 'time' across 'date' first, then split 'time' into 'sdate' and 'time'.**  
Check the two output dimensions of the above script. Are they the same?

3. Comment out `split_multiselected_dims = TRUE` in obs. What is the dimension now?

4. What will happen if `split_multiselected_dims = TRUE` but `merge_across_dims = FALSE`? Does it make sense?


## 4. Others
The parameters we did not mention in this hands-on (but you might use): 'pattern_dims', 'metadata_dims', 'path_glob_permissive'.
You can check the startR document or go to FAQs and usecase on startR GitLab to learn more.
