---
title: "Data Retrieval"
author: "Nicolau Manubens"
date: "`r Sys.Date()`"
output: pdf_document
vignette: >
  %\VignetteEngine{R.rsp::md}
  %\VignetteIndexEntry{Data Retrieval}
  %\usepackage[utf8]{inputenc}
---

Data retrieval
==============
One of the strong points of s2dverification is its data retrieval module.  
Its aim is to load and homogenize forecast or hindcast data generated by 
climate models, usually in multiple runs and multiple ensemble members per 
run, as well as the corresponding observational time series obtained from 
either pure observations or reanalyses.

A climate model run simulates a time period starting at a **starting date** 
and generates data for a series of time steps regularly spaced along the 
**forecast time** that conform the simulated time period. A model run is 
also usually referred to as starting date.  
The **lead-time** at each forecast time step is the amount of simulated time 
since the beginning of the simulation, e.g., the first value of a 
monthly-averaged forecast has a lead-time of 0 months, the second value has a
lead-time of 1 month and so on. In the context of s2dverification, however,
the concept lead-time is used to make reference to the forecast time steps: 
the first lead-time of a forecast is the value with a lead-time of 0 time units 
(the first time step), the second lead-time is the one with a lead-time of 1 
time unit (the second time step) and so on.  
Data at one forecast time (or lead-time) from different runs or starting dates 
conform a time series defined along the **actual time**.  

`Load()` is the interface function to the data retrieval module. Through it 
you can request previously monthly-averaged or daily-averaged 2-dimensional 
variables or global mean variables from as many experimental and observational 
multi-member datasets as needed, at the selected starting dates and forecast 
time steps, over any longitude-latitude region of the Earth surface.

`Load()` automatically fetches the selected starting dates of the requested 
experimental datasets and the date-corresponding observational data, 
homogenizes it and loads it into an R object ready to be processed. 
If done manually this homogenization procedure can be time-consuming and 
become a hurdle in the verification process.

The object returned by `Load()` contains, among others, two arrays with the 
requested data, one for the experimental data and the other for observational 
data, with a disctintive dimension structure of s2dverification:
```r
c(n. of datasets, n. of members, n. of starting dates, n. of lead-times, n. of latitudes, n. of longitudes)
``` 

The homogenization process consists of the following steps:
  1. Regridding data. `Load()` relies on CDO tools for this task (see 
[**Installation**](../README.md)).
  2. Applying masks to disable certain grid cells.
  3. Disabling values beyond certain thresholds.
  4. Computing area-weighted averages.

(The 1st, 2nd and 4th steps only apply if loading 2-dimensional variables.)

Supported file formats
----------------------
There are some guidelines that the files of the datasets must follow so that 
`Load()` is able to recognize, fetch and process them:
  - Data files must be in NetCDF 3/4 local or remote files or indexed in NCML 
catalogs provided by THREDDS servers.
  - Data files must contain monthly or daily time series. Data in other 
frequencies can also be loaded as experimental data, but the automatical fetch
of date-corresponding observations will fail.
  - The only supported source (and target) grids are regular and gaussian grids 
(those definable with the CDO grid names that follow the patterns 'r<NX>x<NY>' 
or 't<RES>grid').
  - The variable of interest must be defined over the dimensions 
(time[, members]) if is a global means time-series or (time[, members], 
latitudes, longitudes) if is a 2-dimensional field. These dimensions 
can be named at will and can be in any order. The less time-consuming orders 
to load are, however, (time[, members]) or (time[, members], latitudes, 
longitudes). Here, the dimension time makes reference to forecast time.
  - If the data is 2-dimensional, a variable with the grid latitudes and a 
variable with the grid longitudes must be defined, with the same name as their
corresponding dimension.
  - The files of an experimental dataset must be distributed in either of the 
following two ways:
    - File per starting date: each file must contain the starting date\* 
somewhere in its path.
    - File per starting date per member: each file must contain the the 
starting date\* and the member number\*\* somewhere in its path.
  - The files of an observational dataset must be distributed in either of the
following three ways:
    - File per month: each file must contain the year and month\*\*\* somewhere 
in its path.
    - File per month per member: each file must contain the member 
number\*\* and the year and month\*\*\* somewhere in its path.
    - Single file: the time axis must be properly defined to work with
`cdo showmon` and `cdo showyear`.

\*: starting dates in the format YYYYMMDD.  
\*\*: member number, from 0 to total_number_of_members - 1, with as many 
preceding 0s as needed to reach the needed number of digits to represent 
total_number_of_members - 1.  
\*\*\*: year in the format YYYY and month in the format MM.  

Load() options
--------------

As of s2dverification v2.5.0, `Load()` has 27 parameters.  
14 of them are for selecting the data of interest, 8 of them are for adjusting
the homogenization steps and the other 5 are for general adjustments of 
`Load()`. See them explained in 3 sections:
  - [**Selection parameters**](#selection)
  - [**Homogenization parameters**](#homogenization)
  - [**General parameters**](#general)

### Selection parameters<a name="selection"></a>
The selection parameters are `var`, `exp`, `obs`, `sdates`, `nmember`, 
`nmemberobs`, `nleadtime` (deprecated), `leadtimemin`, `leadtimemax`, 
`sampleperiod`, `lonmin`, `lonmax`, `latmin` and `latmax`. You can see most of 
them in use in the [**Example**](example.md) except for `nmember`, 
`nmemberobs`, `nleadtime` and `sampleperiod`. The first part of that example 
is reproduced in more detail below in this section.

`var` must be provided with the short name of a 2-dimensional or global mean 
variable of interest, for example `var = 'tas'` (most usually the provided 
name must be the same as the variable name inside the NetCDF data files). 
`exp` and `obs` contain the information on the location of the requested 
datasets, `sdates` and `leadtimemin`/ `leadtimemax` tell `Load()` which 
starting dates and lead-times to pick and so on.  
Check the [**user manual**](../s2dverification-manual.pdf) or `?Load` for a 
thorough explanation.

Now, as an example, let's consider we have two experimental datasets and one
observational datasets, both with data for the variable 'tas' and with the 
files distributed as depicted in the following directory tree:
```
  /path/to/
    |--experiments/
    |    |--experimentA/
    |    |    |--monthly_mean/
    |    |         |--tas/
    |    |              |--tas_19911101.nc
    |    |              |--tas_19921101.nc
    |    |              |        ·
    |    |              |        ·
    |    |              |        ·
    |    |              |--tas_20001101.nc
    |    |--experimentB/
    |         |--monthly_mean/
    |              |--tas/
    |                   |--tas_19911101.nc
    |                   |--tas_19921101.nc
    |                   |        ·
    |                   |        ·
    |                   |        ·
    |                   |--tas_20001101.nc
    |--observations/
         |--observationX/
              |--monthly_mean/
                   |--tas/
                        |--tas_199101.nc
                        |--tas_199102.nc
                        |--tas_199103.nc
                        |--tas_199104.nc
                        |--tas_199105.nc
                        |--tas_199106.nc
                        |--tas_199107.nc
                        |--tas_199108.nc
                        |--tas_199109.nc
                        |--tas_199110.nc
                        |--tas_199111.nc
                        |--tas_199112.nc
                        |--tas_199201.nc
                        |--tas_199202.nc
                        |        ·
                        |        ·
                        |        ·
                        |--tas_199401.nc
                        |--tas_199402.nc
                        |--tas_199403.nc
```
We assume these files are NetCDF 3/4 compliant and have the variable 'tas' 
defined inside, fulfilling the guidelines explained above in the previous 
section.

We can now build the information blocks of each dataset as follows:
```r
library(s2dverification)

expA <- list(
          name = 'experimentA',
          path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc')
        )
expB <- list(
          name = 'experimentB',
          path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc')
        )
obsX <- list(
          name = 'observationX',
          path = file.path('/path/to/observations/$OBS_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$YEAR$$MONTH$.nc')
        )
```
`Load()` will automatically replace the wildcards (keywords starting and
ending with '$') with the corresponding starting dates, variable name, etc. 
See help on parameters `exp` and `obs` in `?Load` for a full explanation.

Now time to issue the `Load()` call: the target variable name will be `'tas'` 
and the parameters `exp` and `obs` can be built from the previous information 
blocks. Apart from that we just need to specify a set of starting dates for 
`Load()` to work:

```r
sdates <- paste0(1991:2000, '1101')
data <- Load(var = 'tas', exp = list(expA, expB), obs = list(obsX), 
             sdates = sdates)
```

This will pick by default as many members and lead-times of each experimental 
dataset as found in the first one ('expA'), and as many members and lead-times 
of each observational dataset (only 1 in this example) as found in the first
one ('obsX').

All the data will be returned in a named list with many components, in a 
format similar to that used in the packages 'downscaleR' and 'loadR' (see 
`?Load` for a full description of the returned metadata). 
The two components that contain the experimental and observational data are
`$mod` and `$obs`. These are two arrays with the dimensions in the following
order:
```r
c(n. of datasets, n. of members, n. of starting dates, n. of lead-times, n. of latitudes, n. of longitudes)
```  
The presence of the last two dimensions will depend on whether the requested
variable is 2-dimensional or global mean and on the selected `output` type 
(see below or in `?Load`). See 
[**Normalizing fields**](statistics.md#normalizing) 
for the *raison d'être* of this array structure for observational data.

In the case of the example, if 'expA' has 5 members but 'expB' has 8, the 
resulting size of the members dimension will be 5 and the last 3 members of
'expB' will be missed. Besides, if 'expB' has only 2 members, the 3 remaining 
members will be filled up with NA values. It happens exactly the same with the
lead-times.  
To avoid this default behaviour you can specify the expected maximum number of 
members in `nmember` and `nmemberobs` and the expected maximum number of 
lead-times in `leadtimemax`.

Also, if not interested in the few first lead-times of the starting dates, you 
can discard them specifying the first lead-time of interest with the parameter
`leadtimemin`, and even pick one lead-time of each `sampleperiod` lead-times 
starting from `leadtimemin`. 

The following code will pick the lead-times corresponding to a time period of 
one year, starting from December, since the first month of the starting dates 
is November and `leadtimemin` is 2:

```r
sdates <- paste0(1991:2000, '1101')
data <- Load(var = 'tas', exp = list(expA, expB), obs = list(obsX), 
             sdates = sdates, leadtimemin = 2, leadtimemax = 13)
```

It is possible to load experimental data only or observational data only by 
providing `NULL` to `exp` or `obs`.

In the example just above, the loaded variable is 'tas', a 2-dimensional 
variable. However though, `Load()` computes by default an area-weighted 
average, hence the component `data$mod` will have dimensions 
`c(2, 5, 3, 6)` and the component `data$obs` will have dimensions 
`c(1, 1, 3, 6)`. Read below in **Homogenization parameters** on how to change 
the `output` type of `Load()`.
 
If willing to work with averages of a specific region of the Earth surface 
you can specify the limits of its bounding box via the parameters `latmin`, 
`latmax`, `lonmin` and `lonmax`. For example, for the North Pacific region:

```r
sdates <- paste0(1991:2000, '1101')
data <- Load(var = 'tas', exp = list(expA, expB), obs = list(obsX), 
             sdates = sdates, leadtimemin = 2, leadtimemax = 13,
             latmin = -10, latmax = 60, lonmin = 100, lonmax = 250)
```

### Homogenization parameters<a name="homogenization"></a>
This group of parameters is conformed by all the parameters that have effects
in any of the homogenization steps mentioned in the introduction of 
[**Data retrieval**](data_retrieval.md) namely: `output`, `method`, `grid`, 
`maskmod`, `maskobs`, `varmin`, `varmax` and `remapcells`.

When loading 2-dimensional variables, `Load()` calculates and returns by 
default area-weighted averages of the values in the grid cells that fall 
inside the requested region. The area averages are computed from the original
datasets in their original grids. `Load()`, however, prior to computing the 
averages, can remap all the data to a common grid if you specify it via the 
`grid` argument. The grid specification must be a character string that follows
one of the patterns 'r<NX>x<NY>' or 't<RES>grid', where '<NX>' and '<NY>' are 
the number of longitudes and latitudes of a rectangular grid and '<RES>' is the
factor of truncation of spherical harmonics of a gaussian grid.  
  E.g.: 'r320x160' or 't73grid'.  

Internally, `Load()` will dispatch CDO calls to interpolate the original files.
One of four methods of interpolation can be chosen through the parameter 
`method`: 'conservative', 'bilinear', 'bicubic' or 'distance-weighted'. The
parameter `remapcells` is related to the precision of grid interpolations, you 
can read more in `?Load`. 

Additionally, any grid cells of your choice can be disabled before computing 
the averages by providing a mask for each dataset in `exp` and `obs`. These 
masks must be provided with the arguments `maskmod` and `maskobs` and can 
simply be a pointer to a NetCDF mask or an R array with the proper size.
Read more on the arguments `maskmod` and `maskobs` in `?Load` to see the 
guidelines the mask files must follow or how to provide a mask as an R array.

It is also possible to work directly with 2-dimensional data without averaging 
by setting the parameter `output = 'lonlat'`. With this option the parameters 
`grid`, `method`, `maskmod` and `maskobs` keep applying: you can request your
data to be in a common rectangular or gaussian grid of your choice and disable
some values with a mask.

To summarize the homogenization parameters mentioned so far, we could dispatch the
following `Load()` call:

```r
mask <- list(path = '/path/to/masks/land_mask_r320x160.nc')
sdates <- paste0(1991:2000, '1101')
data <- Load(var = 'tas', exp = list(expA, expB), obs = list(obsX), 
             sdates = sdates, leadtimemin = 2, leadtimemax = 13,
             latmin = -10, latmax = 60, lonmin = 100, lonmax = 250, 
             output = 'lonlat', grid = 'r320x160', method = 'bicubic',
             maskmod = list(mask, mask), maskobs = list(mask))
```

Finally, extreme values can be disabled by setting a upper and a lower 
threshold via the parameters `varmin` and `varmax`.

### General parameters<a name="general"></a>
The general parameters are `storefreq`, `configfile`, `silent`, `nprocs` and 
`dimnames`.

#### Loading daily data
It is possible to load not only monthly data but also daily data with exactly
the same funcionalities available in both modes. `storefreq` tells `Load()` 
whether the experimental data to load is stored in a monthly frequency or in a 
daily frequency. This way it will be able to work out the dates of the 
corresponding observational data and fetch the corresponding files. The only 
supported values for this parameter currently are 'monthly' (the default) and 
'daily'.  
Note that in the case that only experimental datasets are requested 
(via `exp`) any storage frequency is be supported, since `Load()` doesn't 
have to make date-correspondences. It is the user's responsibility in these
cases to ensure that the experimental datasets are stored in a common time 
frequency.

#### Adjusting dimension names
Regarding the guidelines that the files of a dataset must follow, the variable
of interest inside the NetCDF files must be defined over the dimensions 
(time[, members][, latitudes, longitudes]). These dimensions can be named at 
will but, by default, the expected names are 'ensemble', 'latitude' 
and 'longitude' (the time dimension can take any name). If the name of any of 
the dimensions in all the datasets to load is different to this default, it 
can be adjusted with the parameter `dimnames` as in the following example:
```r
dimnames = list(member = 'members', lat = 'j', lon = 'i')
```
However, if only one of the datasets breaks the rules or each dataset has its
own naming convention, you can adjust individually when building the 
information blocks of each dataset as in the following example:
```r
expA <- list(
          name = 'experimentA',
          path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc'),
          dimnames = list(member = 'members')
        )
expB <- list(
          name = 'experimentB',
          path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc'),
          dimnames = list(lat = 'j', lon = 'i')
        )
obsX <- list(
          name = 'observationX',
          path = file.path('/path/to/observations/$OBS_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$YEAR$$MONTH$.nc'),
          dimnames = list(member = 'ens', lat = 'y', lon = 'x')
        )
```

#### Workflow of `Load()` and parallelism
The workflow of `Load()` consists of 3 steps:
  1. Building up work pieces
Each work piece is a packet of information that describes the location of a 
**single** data file requested in a `Load()` call (e.g. 'tas_19901101.nc') and
the homogenization and arranging operations to be applied to that file.
  2. Setting up a cluster
A multi-core cluster is set up. As of now `Load()` can only make the most of
multiple cores in a single processor but not of multiple processors.
The number of nodes in the multi-core cluster can be adjusted via `nprocs` 
(number of processes, as each node is a full new R process). Note that the 
number of nodes does not necessarly have to coincide with the number of cores:
if `nprocs` is lower than the number of cores, probably some of the available
cores will remain unused and if `nprocs` is higher, a single core will have 
more than one task assigned at a time. Depending on the environment this may 
be convenient.  
Three aspects to take into account when selecting the appropriate `nprocs` are: 
    - Whether hyperthreading is activated in the system
    - The fact that each node will dispatch parallel blocking calls to CDO 
along its execution
    - A 'bigmemory' shared memory region will be set up for the multiple cores 
to arrange the resulting data of each work piece.  
Setting `nprocs = 1` will avoid creating a cluster and all the workload will
be carried out in a single process (that may dispatch parallel blocking CDO
calls).
  3. Dispatching the work pieces to the cluster
Once the work pieces and the cluster are ready, `Load()` dispatches the work
pieces with load balancing to the nodes of the cluster. Each node will keep
applying the internal function `LoadDataFile()` to each work piece.

#### Running `Load()` silently
All along the process `Load()` keeps informing about the status and any 
detected potential issues. When run in batch mode it may be of your interest
to turn the messages off by setting `silent = TRUE`.

#### Using a configuration file
When using `Load()` frequently, one may end up repeating many times the code 
to build the same information blocks to load certain specific datasets.
Besides, when using `Load()` in a team, one of its members may be interested 
in using datasets that other members usually work with or may want to make 
accessible his datasets to the colleagues.

To help in these two situations `Load()` has a built in mechanism to read 
**configuration files** that contain, in a tabular and compact format, the 
information blocks of the datasets of interest. This way, by defining the 
information of a dataset only once in a configuration file, `Load()` will be
able to read its information as needed. Also, the members of a team can share
a configuration file to put the information of their datasets at others'
disposal.

A configuration file can be created by hand, following the specifications in
`?ConfigFileOpen`, or can be created and edited through a set of functions 
included in s2dverification: `ConfigFileCreate()`, `ConfigFileOpen()`, 
`ConfigAddEntry()`, `ConfigEditEntry()`, `ConfigRemoveEntry()`, 
`ConfigEditDefinition()`, `ConfigRemoveDefinition()`, `ConfigFileSave()`, 
`ConfigShowTable()`, `ConfigShowDefinitions()`, `ConfigShowSimilarEntries()` 
and `ConfigApplyMatchingEntries()`.  

The first option can be faster for experienced users but the second option is 
safer and useful to automatise the creation of configuration files. The 
second option will be used in the following examples but the raw file will 
also be displayed to show the internals of the configuration files and to 
give some hints on how to create them manually.

A blank new configuration file can be created as follows:
```r
conf_path <- '/path/to/configfile.conf'
ConfigFileCreate(conf_path)
```
This will generate a file at the specified path with all the boilerplate:
```
# s2dverification configuration file
#
# Check ?ConfigFileOpen after loading s2dverification for detailed 
# documentation on this configuration file.

#############
!!definitions
#############
DEFAULT_EXP_MAIN_PATH = $EXP_NAME$
DEFAULT_EXP_FILE_PATH = $STORE_FREQ$/$VAR_NAME$_$START_DATE$.nc
DEFAULT_NC_VAR_NAME = $VAR_NAME$
DEFAULT_SUFFIX =
DEFAULT_VAR_MIN =
DEFAULT_VAR_MAX =
DEFAULT_OBS_MAIN_PATH = $OBS_NAME$
DEFAULT_OBS_FILE_PATH = $STORE_FREQ$/$VAR_NAME$_$YEAR$$MONTH$.nc
DEFAULT_DIM_NAME_LONGITUDES = longitude
DEFAULT_DIM_NAME_LATITUDES = latitude
DEFAULT_DIM_NAME_MEMBERS = ensemble


######################
!!table of experiments
######################
#exp_name, var_name[, exp_main_path[, exp_file_path[, nc_var_name[, suffix[, var_min[, var_max]]]]]]

#######################
!!table of observations
#######################
#obs_name, var_name[, obs_main_path[, obs_file_path[, nc_var_name[, suffix[, var_min[, var_max]]]]]]
```
The lines starting with '#' are comments and the ones starting with '!' are 
key lines that can't be moved or removed.

As just shown, the configuration file consists of a list of definitions and of 
two tables, one to keep information on experimental datasets and the other
for observational datasets.

The rows or entries of the tables will be filled up each with information 
relative to a dataset. An entry will associate a pair of dataset name and 
variable name to a set of information pieces.  
Each entry keeps 8 columns, the pair of keys and 6 information pieces 
associated to a dataset: the dataset short name or identifier, the variable 
short name, the main path or path to the dataset main folder, the file path or 
path to the NetCDF files of the dataset, the actual variable name inside the 
files, a suffix (read further) and a lower and uper thresholds to disable 
values of the dataset beyond them.  
The list of definitions contains, initially, the default values for the columns
of the tables. It can be extended with other definitions that can be accessed 
from throughout the configuration file, which is useful to avoid repeating 
certain paths, for example, in multiple entries. A definition follows the 
format `VARIABLE = value` and its value can be invoked with `$VARIABLE$`.   
Apart from these explicit definitions, there are some implicit key definitions:
  - `$EXP_NAME$` to retrieve, from an experimental dataset entry, the 
identifier name of an experimenal dataset, as requested to `Load()`.
  - `$OBS_NAME$`, similar to `$OBS_NAME$`.
  - `$VAR_NAME$`, the short name of the variable, as requested to `Load()`.
  - `$START_DATE$`, one of the starting dates, as requested to `Load()`.
  - `$YEAR$`, `$MONTH$`, `$DAY$`, taken from the `$START_DATE$`.
  - `$MEMBER_NUMBER$`, the number of the member in loadin process, from 0 to 
total_number_of_members - 1 and with padding zeros.
  - `$STORE_FREQ$`, the frequency of the experimental data in disk, as 
specified to `Load()` via `storefreq`.

The next step is to add the entries of the datasets of interest. It can be 
achieved with a combination of the functions `ConfigFileOpen()`, 
`ConfigAddEntry()` and `ConfigFileSave()`:
```r
c <- ConfigFileOpen(conf_path)
c <- ConfigAddEntry(c, 'experiments', dataset_name = 'expA', var_name = 'tas',
                    main_path = '/path/to/experiments/$EXP_NAME$/',
                    file_path = 'monthly_mean/$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc')
c <- ConfigAddEntry(c, 'experiments', dataset_name = 'expB', var_name = 'tas',
                    main_path = '/path/to/experiments/$EXP_NAME$/',
                    file_path = 'monthly_mean/$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc')
c <- ConfigAddEntry(c, 'observations', dataset_name = 'obsX', var_name = 'tas',
                    main_path = '/path/to/observations/$OBS_NAME$/',
                    file_path = 'monthly_mean/$VAR_NAME$/$VAR_NAME$_$YEAR$$MONTH$.nc')
ConfigFileSave(c, conf_path)
```

The tables in the file, at this point, are shown next:
```
######################
!!table of experiments
######################
#exp_name, var_name[, exp_main_path[, exp_file_path[, nc_var_name[, suffix[, var_min[, var_max]]]]]]
expA, tas, /path/to/experiments/$EXP_NAME$/, monthly_mean/$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc, *, *, *, *
expB, tas, /path/to/experiments/$EXP_NAME$/, monthly_mean/$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc, *, *, *, *

#######################
!!table of observations
#######################
#obs_name, var_name[, obs_main_path[, obs_file_path[, nc_var_name[, suffix[, var_min[, var_max]]]]]]
obsX, tas, /path/to/observations/$OBS_NAME$/, monthly_mean/$VAR_NAME$/$VAR_NAME$_$YEAR$$MONTH$.nc, *, *, *, *
```

An asterisk ('*') means that the cell will take the default value defined in 
the defaults for the corresponding column. So the column 'nc_var_name' takes
'$VAR_NAME$' which is then replaced by 'tas' in this case. The other default
values are all empty.

Once the file is ready it can be used from `Load()` by specifying its path via
the parameter `configfile`. Then, `exp` and `obs` can be used in its simple
format: a vector of character strings that tell the name of the datasets of
interest. For example:
```r
sdates <- paste0(1991:2000, '1101')
data <- Load(var = 'tas', exp = c('expA', 'expB'), obs = c('obsX'),
             sdates = sdates, configfile = '/path/to/configfile.conf')
```

From this point on, one could create files with entries for each pair of 
(dataset name, variable name) of interest.

However, the first two columns of the tables in a configuration file can also 
take regular expressions to match more than one dataset name or variable name. 
This way, a more compact and generalized table can be obtained:
```r
c <- ConfigFileOpen(conf_path)
c <- ConfigAddEntry(c, 'experiments', dataset_name = '.*', var_name = '.*',
                    main_path = '/path/to/experiments/$EXP_NAME$/',
                    file_path = 'monthly_mean/$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc')
c <- ConfigAddEntry(c, 'observations', dataset_name = '.*', var_name = '.*',
                    main_path = '/path/to/observations/$OBS_NAME$/',
                    file_path = 'monthly_mean/$VAR_NAME$/$VAR_NAME$_$YEAR$$MONTH$.nc')
ConfigFileSave(c, conf_path)
```

The corresponding tables are:
```
######################
!!table of experiments
###################### 
#exp_name, var_name[, exp_main_path[, exp_file_path[, nc_var_name[, suffix[, var_min[, var_max]]]]]]
.*, .*, /path/to/experiments/$EXP_NAME$/, monthly_mean/$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc, *, *, *, *

#######################
!!table of observations
#######################
#obs_name, var_name[, obs_main_path[, obs_file_path[, nc_var_name[, suffix[, var_min[, var_max]]]]]]
.*, .*, /path/to/observations/$OBS_NAME$/, monthly_mean/$VAR_NAME$/$VAR_NAME$_$YEAR$$MONTH$.nc, *, *, *, *
```

The regular expression '.*' matches any sequence of characters, so these 
entries will assign to all the (experiment name, variable name) pairs a path 
with the pattern  
`/path/to/experiments/$EXP_NAME$/monthly_mean/$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc`  
or, to all the (observation name, variable name) pairs a path with the pattern  
`/path/to/observations/$OBS_NAME$/monthly_mean/$VAR_NAME$/$VAR_NAME$_$YEAR$$MONTH$.nc`.  
In other words, with only two entries in the configuration file you will be 
able to load many variables from many datasets, as long as they follow a 
common convention.

If more than one entry match a pair (dataset name, variable name) all of 
them will be applied from more general to more specific to obtain a value
for each of the 6 information pieces that, at the end, will tell the location 
and information of the requested variable for the requested dataset.

The configuration file, all in all, is a complex but powerful mechanism.
Read `?ConfigFileOpen` for a complete explanation of the details of the tables 
in the configuration files and how multiple matches are applied to build up the
final information. Also see there why the 'suffix' column can be useful.

#### Re-arranging long runs
Sometimes it can happen that one needs to compare data from two experimental 
datasets, one stored in multiple files (one per starting dates) and the other
stored in a single long run (one file with one starting date).

In this case running one single `Load()` call won't be enough since the 
`sdates` argument has to take different values in function of the distribution
of the starting dates of the dataset to load. Besides, running two separate 
`Load()` calls with different starting dates yields two non-comparable data 
sets.

To solve this, it is possible to re-arrange the long run into multiple starting
dates with `Histo2Hindcast()` by just specifying the original starting date,
the series of target starting dates and the number of target lead-times:

Loading the multiple run experiment:
``` 
expA <- list(
          name = 'experimentA',
          path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc')
        )
sdatesA <- paste0(1991:2000, '1101')
leadtimesA <- 12
data <- Load(var = 'tas', exp = list(expA), obs = NULL, sdates = sdatesA,
             leadtimemax = leadtimesA)
```

Loading the single run experiment:
```
expC <- list(
          name = 'experimentC',
          path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                           '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc')
        )
sdatesC <- '19501101'
leadtimemin <- (1991 - 1950) * 12 + 1
leadtimemax <- leadtimemin + (2000 - 1991 + 1) * 12
data <- Load(var = 'tas', exp = list(expC), obs = NULL, sdates = sdatesC,
             leadtimemin = leadtimemin, leadtimemax = leadtimemax)
data <- Histo2Hindcast(data, sdatesA[1], sdatesA, leadtimesA)
```
