Data retrieval
==============
One of the strong points of s2dverification is its data retrieval module.  
`Load()` is the interface function to this module. Through it you can request 
monthly or daily global mean or 2-dimensional variables from as many 
experimental and observational multi-member datasets as needed, at the 
selected starting dates and leadtimes, over any region of interest of the 
Earth surface.

`Load()` automatically fetches the starting dates of the requested 
experimental datasets and the date-corresponding observational data, 
homogenizes it and loads it into an R object. If done manually, this task can 
be time-consuming and become a hurdle in the verification process.

The homogenization consists of the following steps:
  1. Regridding data. `Load()` relies on CDO tools for this task (see 
[**Installation**](../README.md)).
  2. Applying masks to disable certain grid cells.
  3. Disabling values beyond certain thresholds.
  4. Computing area-weighted averages.

The 1st, 2nd and 4th steps only apply if loading 2-dimensional variables.

Supported file formats
----------------------

There are some guidelines that the files of the datasets must follow so that 
`Load()` is able to recognize, fetch and process them:
  - Data files must be in NetCDF 3/4 local or remote files or indexed in NCML 
catalogs provided by THREDDS servers.
  - Data files must contain monthly or daily time series. Data in other 
frequencies can also be loaded as experimental data, but the automatical fetch
of date-corresponding observations will fail.
  - The only supported source (and target) grids are regular and gaussian grids 
(those definable with the CDO grid names that follow the patterns 'r<NX>x<NY>' 
or 't<RES>grid').
  - The variable of interest must be defined over the dimensions 
(time[, members]) if is a global means time-series or (time[, members], 
latitudes, longitudes) if is a 2-dimensional field. These dimensions 
can be named at will and can be in any order. The less time-consuming orders 
to load are, however, (time[, members]) or (time[, members], latitudes, 
longitudes).
  - If the data is 2-dimensional, a variable with the grid latitudes and a 
variable with the grid longitudes must be defined, with the same name as their
corresponding dimension.
  - The files of an experimental dataset must be distributed in either of the 
following two ways:
    - File per starting date: each file must contain the starting date\* 
somewhere in its path.
    - File per starting date per member: each file must contain the the 
starting date\* and the member number\*\* somewhere in its path.
  - The files of an observational dataset must be distributed in either of the
following three ways:
    - File per month: each file must contain the year and month\*\*\* somewhere 
in its path.
    - File per month per member: each file must contain the member 
number\*\* and the year and month\*\*\* somewhere in its path.
    - Single file: the time axis must be properly defined to work with
`cdo showmon` and `cdo showyear`.

\*: starting dates in the format YYYYMMDD.  
\*\*: member number, from 0 to total_number_of_members - 1, with as many 
preceding 0s as needed to reach the needed number of digits to represent 
total_number_of_members - 1.  
\*\*\*: year in the format YYYY and month in the format MM.  

Load() options
--------------

As of s2dverification v2.5.0, `Load()` has 27 parameters.  
14 of them are for selecting the data of interest, 8 of them are for adjusting
the homogenization steps and the other 5 are for general adjustments of 
`Load()`.

### Selection parameters

The selection parameters are `var`, `exp`, `obs`, `sdates`, `nmember`, 
`nmemberobs`, `nleadtime` (deprecated), `leadtimemin`, `leadtimemax`, 
`sampleperiod`, `lonmin`, `lonmax`, `latmin` and `latmax`. You can see most of 
them in use in the [**Example**](example.md) except for `nmember`, 
`nmemberobs`, `nleadtime` and `sampleperiod`. The first part of that example 
is reproduced in more detail below in this section.

`var` must be provided with the short name of a 2-dimensional or global mean 
variable of interest, for example `var = 'tas'` (most usually the provided 
name must be the same as the variable name inside the NetCDF data files). 
`exp` and `obs` contain the information on the location of the requested 
datasets, `sdates` and `leadtimemin`/ `leadtimemax` tell `Load()` which 
starting dates and leadtimes to pick and so on.  
Check the [**user manual**](../s2dverification-manual.pdf) or `?Load` for a 
thorough explanation.

Now, as an example, let's consider we have two experimental datasets and one
observational datasets, both with data for the variable 'tas' and with the 
files distributed as depicted in the following directory tree:
```
  /path/to/
    |--experiments/
    |    |--experimentA/
    |    |    |--monthly_mean/
    |    |         |--tas/
    |    |              |--tas_19911101.nc
    |    |              |--tas_19921101.nc
    |    |              |--tas_19931101.nc
    |    |--experimentB/
    |         |--monthly_mean/
    |              |--tas/
    |                   |--tas_19911101.nc
    |                   |--tas_19921101.nc
    |                   |--tas_19931101.nc
    |--observations/
         |--observationX/
              |--monthly_mean/
                   |--tas/
                        |--tas_199101.nc
                        |--tas_199102.nc
                        |--tas_199103.nc
                        |--tas_199104.nc
                        |--tas_199105.nc
                        |--tas_199106.nc
                        |--tas_199107.nc
                        |--tas_199108.nc
                        |--tas_199109.nc
                        |--tas_199110.nc
                        |--tas_199111.nc
                        |--tas_199112.nc
                        |--tas_199201.nc
                        |--tas_199202.nc
                        |        ·
                        |        ·
                        |        ·
                        |--tas_199401.nc
                        |--tas_199402.nc
                        |--tas_199403.nc
```
We assume these files are NetCDF 3/4 compliant and have the variable 'tas' 
defined inside, fulfilling the guidelines explained above in the previous 
section.

We can now build the information blocks of each dataset as follows:
```r
expA <- list(
          name = 'experimentA',
          path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc')
        )
expB <- list(
          name = 'experimentB',
          path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc')
        )
obsX <- list(
          name = 'observationX',
          path = file.path('/path/to/observations/$OBS_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$YEAR$$MONTH$.nc')
        )
```
`Load()` will automatically replace the wildcards (keywords starting and
ending with '$') with the corresponding starting dates, variable name, etc. 
See help on parameters `exp` and `obs` in `?Load` for a full explanation.

Now time to issue the `Load()` call: the target variable name will be `'tas'` 
and the parameters `exp` and `obs` can be built from the previous information 
blocks. Apart from that we just need to specify a set of starting dates for 
`Load()` to work:

```r
sdates <- paste0(1991:1993, '1101')
data <- Load(var = 'tas', exp = list(expA, expB), obs = list(obsX), 
             sdates = sdates)
```

This will pick by default as many members and leadtimes of each experimental 
dataset as found in the first one ('expA'), and as many members and leadtimes 
of each observational dataset (only 1 in this example) as found in the first
one ('obsX').

All the data will be returned in a named list with many components, in a 
format similar to that used in the packages 'downscaleR' and 'loadR'. 
The two components that contain the experimental and observational data are
`$mod` and `$obs`. These are two arrays with the dimensions in the following
order:
```r
c(n. of datasets, n. of members, n. of starting dates, n. of leadtimes, n. of latitudes, n. of longitudes)
```  
The presence of the last two dimensions will depend on whether the requested
variable is 2-dimensional or global mean and on the selected `output` type 
(see below or in `?Load`).

In the case of the example, if 'expA' has 5 members but 'expB' has 8, the 
resulting size of the members dimension will be 5 and the last 3 members of
'expB' will be missed. Besides, if 'expB' has only 2 members, the 3 remaining 
members will be filled up with NA values. It happens exactly the same with the
leadtimes.  
To avoid this default behaviour you can specify the expected maximum number of 
members in `nmember` and `nmemberobs` and the expected maximum number of 
leadtimes in `leadtimemax`.

Also, if not interested in the few first leadtimes of the starting dates, you 
can discard them specifying the first leadtime of interest with the parameter
`leadtimemin`, and even pick one leadtime of each `sampleperiod`leadtimes 
starting from `leadtimemin`. 

The following code will pick the leadtimes corresponding to the seasons 'DJF' 
and 'MAM', since the first month of the starting dates is November:

```r
sdates <- paste0(1991:1993, '1101')
data <- Load(var = 'tas', exp = list(expA, expB), obs = list(obsX), 
             sdates = sdates, leadtimemin = 2, leadtimemax = 7)
```

It is possible to load experimental data only or observational data only by 
providing `NULL` to `exp` or `obs`.

In the example just above, the loaded variable is 'tas', a 2-dimensional 
variable. However though, `Load()` computes by default an area-weighted 
average, hence the component `data$mod` will have dimensions 
`c(2, 5, 3, 6)` and the component `data$obs` will have dimensions 
`c(1, 1, 3, 6)`. Read below in **Homogenization parameters** on how to change 
the `output` type of `Load()`.
 
If willing to work with averages of a specific region of the Earth surface 
you can specify the limits of its bounding box via the parameters `latmin`, 
`latmax`, `lonmin` and `lonmax`. For example, for the Atlantic region:

```r
sdates <- paste0(1991:1993, '1101')
data <- Load(var = 'tas', exp = list(expA, expB), obs = list(obsX), 
             sdates = sdates, leadtimemin = 2, leadtimemax = 7,
             latmin = -10, latmax = 60, lonmin = 100, lonmax = 250)
```

### Homogenization parameters

This group of parameters is conformed by all the parameters that have effects
in any of the homogenization steps mentioned in the introduction of 
[**Data retrieval**](data_retrieval.md) namely: `output`, `method`, `grid`, 
`maskmod`, `maskobs`, `varmin`, `varmax` and `remapcells`.

When loading 2-dimensional variables, `Load()` calculates and returns by 
default area-weighted averages of the values in the grid cells that fall 
inside the requested region. The area averages are computed from the original
datasets in their original grids. `Load()`, however, prior to computing the 
averages, can remap all the data to a common grid if you specify it via the 
`grid` argument. The grid specification must be a character string that follows
one of the patterns 'r<NX>x<NY>' or 't<RES>grid', where '<NX>' and '<NY>' are 
the number of longitudes and latitudes of a rectangular grid and '<RES>' is the
factor of truncation of spherical harmonics of a gaussian grid.  
  E.g.: 'r320x160' or 't73grid'.  

Internally, `Load()` will dispatch CDO calls to interpolate the original files.
One of four methods of interpolation can be chosen through the parameter 
`method`: 'conservative', 'bilinear', 'bicubic' or 'distance-weighted'. The
parameter `remapcells` is related to the precision of grid interpolations, you 
can read more in `?Load`. 

Additionally, any grid cells of your choice can be disabled before computing 
the averages by providing a mask for each dataset in `exp` and `obs`. These 
masks must be provided with the arguments `maskmod` and `maskobs` and can 
simply be a pointer to a NetCDF mask or an R array with the proper size.
Read more on the arguments `maskmod` and `maskobs` in `?Load` to see the 
guidelines the mask files must follow or how to provide a mask as an R array.

It is also possible to work directly with 2-dimensional data without averaging 
by setting the parameter `output = 'lonlat'`. With this option the parameters 
`grid`, `method`, `maskmod` and `maskobs` keep applying: you can request your
data to be in a common rectangular or gaussian grid of your choice and disable
some values with a mask.

To summarize the homogenization parameters mentioned so far, we could dispatch the
following `Load()` call:

```r
mask <- list(path = '/path/to/masks/land_mask_r320x160.nc')
sdates <- paste0(1991:1993, '1101')
data <- Load(var = 'tas', exp = list(expA, expB), obs = list(obsX), 
             sdates = sdates, leadtimemin = 2, leadtimemax = 7,
             latmin = -10, latmax = 60, lonmin = 100, lonmax = 250, 
             output = 'lonlat', grid = 'r320x160', method = 'bicubic',
             maskmod = list(mask, mask), maskobs = list(mask))
```

Finally, extreme values can be disabled by setting a upper and a lower 
threshold via the parameters `varmin` and `varmax`.

### General parameters

The general parameters are `storefreq`, `configfile`, `silent`, `nprocs` and 
`dimnames`.

#### Loading daily data
It is possible to load not only monthly data but also daily data with exactly
the same funcionalities available in both modes. `storefreq` tells `Load()` 
whether the experimental data to load is stored in a monthly frequency or in a 
daily frequency. This way it will be able to work out the dates of the 
corresponding observational data and fetch the corresponding files. The only 
supported values for this parameter currently are 'monthly' (the default) and 
'daily'.  
Note that in the case that only experimental datasets are requested 
(via `exp`) any storage frequency is be supported, since `Load()` doesn't 
have to make date-correspondences. It is the user's responsibility in these
cases to ensure that the experimental datasets are stored in a common time 
frequency.

#### Adjusting dimension names
Regarding the guidelines that the files of a dataset must follow, the variable
of interest inside the NetCDF files must be defined over the dimensions 
(time[, members][, latitudes, longitudes]). These dimensions can be named at 
will but, by default, the expected names are 'ensemble', 'latitude' 
and 'longitude' (the time dimension can take any name). If the name of any of 
the dimensions in all the datasets to load is different to this default, it 
can be adjusted with the parameter `dimnames` as in the following example:
```r
`dimnames = list(member = 'members', lat = 'j', lon = 'i')`
```
However, if only one of the datasets breaks the rules or each dataset has its
own naming convention, you can adjust individually when building the 
information blocks of each dataset as in the following example:
```r
expA <- list(
          name = 'experimentA',
          path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc'),
          dimnames = list(member = 'members')
        )
expB <- list(
          name = 'experimentB',
          path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc'),
          dimnames = list(lat = 'j', lon = 'i')
        )
obsX <- list(
          name = 'observationX',
          path = file.path('/path/to/observations/$OBS_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$YEAR$$MONTH$.nc'),
          dimnames = list(member = 'ens', lat = 'y', lon = 'x')
        )
```

#### Workflow of `Load()` and parallelism
The workflow of `Load()` consists of 3 steps:
  1. Building up work pieces
Each work piece is a packet of information that describes the location of a 
**single** data file requested in a `Load()` call (e.g. 'tas_19901101.nc') and
the homogenization and arranging operations to be applied to that file.
  2. Setting up a cluster
A multi-core cluster is set up. As of now `Load()` can only make the most of
multiple cores in a single processor but not of multiple processors.
The number of nodes in the multi-core cluster can be adjusted via `nprocs` 
(number of processes, as each node is a full new R process). Note that the 
number of nodes does not necessarly have to coincide with the number of cores:
if `nprocs` is lower than the number of cores, probably some of the available
cores will remain unused and if `nprocs` is higher, a single core will have 
more than one task assigned at a time. Depending on the environment this may 
be convenient.  
Three aspects to take into account when selecting the appropriate `nprocs` are 
a) whether hyperthreading is activated in the system, b) the fact that each 
node will dispatch parallel blocking calls to CDO along ints execution and c)
that a 'bigmemory' shared memory region will be set up for the multiple cores 
to arrange the resulting data of each work piece.  
Setting `nprocs = 1` will avoid creating a cluster and all the workload will
be carried out in a single process (that may dispatch parallel blocking CDO
calls).
  3. Dispatching the work pieces to the cluster
Once the work pieces and the cluster are ready, `Load()` dispatches the work
pieces with load balancing to the nodes of the cluster. Each node will keep
applying the internal function `LoadDataFile()` to each work piece.

#### Running `Load()` silently

All along the process `Load()` keeps informing about the status and any 
detected potential issues. When run in batch mode it may be of your interest
to turn the messages off by setting `silent = TRUE`.

#### Using a configuration file
here at some point explain that one can use suffix when defining the exp and obs object!!).
