Statistics
==========
The **Statistics** module consists of a set of functions that are commonly 
used in the forecast verification process to modify or describe series of 
modelled or observed measurements for a certain physic variable of interest, or 
**fields**, and some functions to generate synthetic data that complies with 
a given set of distribution parameters.

These functions can be classified in 5 groups as a function of their purpose:

  - [**Normalizing fields**](#normalizing): `Clim()`, `Ano()` and 
`Ano_Crossvalid()`.
  - [**Describing fields**](#describing): `Trend()`, `Consist_Trend()`, 
`Regression()`, `Alpha()`, `FitAcfCoef()`, `FitAutocor()` and `Spectrum()`.
  - [**Filtering fields**](#filtering): `Trend()`, `Consist_Trend()`, 
`Regression()`, `Smoothing()` and `Filter()`.
  - [**Generating derivative fields**](#generating_der): `Season()`, 
`StatSeasAtlHurr()` and `ProbBins()`.
  - [**Generating new fields**](#generating_new): `GenSeries()`, `Toymodel()`, 
`Eno()` and `EnoNew()`.

It is important, to master these functions, to have in mind the structure and 
meaning of the dimensions of the data arrays provided by the 
[**Data retrieval**](data_retrieval.md) module used throughout in 
s2dverification:
```r
  c(n. of datasets, n. of members, n. of starting dates, n. of lead-times, n. of latitudes, n. of longitudes)
```
the last two being present in function of the parameter `output`.

The explanations in the following sections will be based on the same data used 
in the [**Example**](example.md). See [**Data retrieval**](data_retrieval.md) 
for details.
```r
library(s2dverification)
expA <- list(name = 'experimentA',
             path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                              '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc'))
expB <- list(name = 'experimentB',
             path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                              '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc'))
obsX <- list(name = 'observationX',
             path = file.path('/path/to/observations/$OBS_NAME$/monthly_mean',
                              '$VAR_NAME$/$VAR_NAME$_$YEAR$$MONTH$.nc'))
sdates <- paste0(1991:1993, '1101')
data <- Load('tas', list(expA, expB), list(obsX),
             sdates = sdates,
             leadtimemin = 2, leadtimemax = 7,
             latmin = -10, latmax = 60,
             lonmin = 100, lonmax = 250,
             output = 'areave', grid = 't106grid',
             method = 'distance-weighted')
```

Normalizing fields<a name="normalizing"></a>
------------------
Due to systematic errors in the models their mean behaviour along the forecast
time can differ or depart from the mean behaviour of the observed data over 
that forecast time period, a phenomenon also known as **drift**.  
It is hence a standard practice in climate forecast verification to compare 
only the deviations of the experimental and observational data from their mean 
behaviour along the forecast time. For this reason most of the functions in 
s2dverification use series of deviations as inputs and provide series of 
deviations as outputs.

The mean behaviour along the forecast time of a data set, also known as the 
**climatology**, can be computed with multiple methods. The deviation from 
the climatology is known as the **anomaly** or, more explicitly, 
drift-corrected anomaly, and it can be calculated by simply subtracting the 
climatology of a data set to its raw data.

A general purpose climatology could be computed as follows (the 3rd dimension 
corresponds to the starting dates or model runs):
```r
clim_exp <- apply(data$mod, dim(data$mod)[-3], mean, na.rm = TRUE)
```
or with the function `Mean1Dim()` in s2dverification:
```r
clim_exp <- Mean1Dim(data$mod, 3)
```

s2dverification, however, builds upon the per-pair method to compute 
climatologies. It is implemented in the function `Clim()`.  
Its parameters `kharin` and `NDV` allow to select two other more refined 
methods: the trend method and the initial-conditions method. All of them are 
documented in
```
Fučkar N, Volpi D, Guemas V, Doblas-Reyes F, 2014, A posteriori adjustment of near-term climate predictions: Accounting for the drift dependence on the initial conditions. Geophysical Research Letters, 41 (14), 5200–5207, doi:10.1002/2014GL060815
```
Following the per-pair method the climatology of an experimental dataset is 
obtained, as in the general purpose climatology, by calculating an average 
across the starting dates at each forecast time, yielding a time series 
defined along the forecast times of the experiment. The climatology of an 
observational dataset is obtained by computing, for each forecast time, the 
average of the observations that date-correspond that leadtime of each 
starting date.  
To make easy this process, `Load()` arranges both experimental and 
observational data in a forecast-like arrangement.

`Clim()` and the per-pair method only account, at each forecast time, for the 
starting dates for which data is available in all the experimental and 
observational datasets, hence both experimental and observational data must be
provided as inputs:

```r
clim <- Clim(data$mod, data$obs)
```

The raw data and the climatologies can be visualized with `PlotAno()` and 
`PlotClim()`, respectively:
```r
selected_sdates <- gsub('1101', '1201', sdates)
PlotAno(data$mod, data$obs, selected_sdates, 
        toptitle = paste0(c("Experiment A", "Experiment B"), 
                          ": Raw 'tas' over Atlantic region."), 
        ytitle = c("K", "K"), 
        fileout = paste0("stat_raw_exp", c("A", "B"), "_obsX.eps")) 
PlotClim(clim$clim_exp, clim$clim_obs, monini = 12,
         toptitle = "Per-pair 'tas' climatologies over the Atlantic region.", 
         ytitle = "K",
         listexp = c('Experiment A', 'Experiment B'),
         listobs = c('Observation X'),
         fileout = "stat_clim_expA_expB_obsX.eps")
```
<img src="stat_raw_expA_obsX.png" width="600" />
<img src="stat_raw_expB_obsX.png" width="600" /> 
<img src="stat_clim_expA_expB_obsX.png" width="600" /> 

Each plot in the `Clim()` figure corresponds to the climatology of a member of 
the corresponding experiment or observation. Usually, if working with 
atmospheric variables, it is not relevant when computing climatologies to make 
a distinction among members because they are considered to be equivalent. It 
is possible, after seeing they are indeed equivalent, to compute the ensemble
mean climatology with
```r
exp_ens_clim <- Mean1Dim(clim$clim_exp, 2)
obs_ens_clim <- Mean1Dim(clim$clim_obs, 2)
```
or, if sure a priori of their equivalence, by setting the parameter 
`memb = FALSE` in `Clim()`: 
```r
clim <- Clim(data$mod, data$obs, memb = FALSE)
```

If working with oceanic variables this may not necessarly apply. See
```
Du, H., F.J. Doblas-Reyes, J. García-Serrano, V. Guemas, Y. Soufflet and B. Wouters (2012). Sensitivity of decadal predictions to the initial atmospheric and oceanic perturbations. Climate Dynamics, 39, 2013-2023, doi:10.1007/s00382-011-1285-9. http://link.springer.com/article/10.1007%2Fs00382-011-1285-9 Fig 2d is a very good example. If only one climatology is used for this variable, the skill is substantially lower.
```

Once the climatologies are computed, the anomalies can be obtained with 
`Ano()`, that subtracts the climatologies to each member, starting date,
latitude and longitude of their corresponding dataset:
```r  
ano_exp <- Ano(data$mod, clim$clim_exp)
ano_obs <- Ano(data$obs, clim$clim_obs)
```

All the starting dates of an experiment contribute to the per-pair climatology 
and when computing the anomaly of that starting date its contribution to the 
climatology is subtracted, which can imply a loss of information [citation]. 
To avoid this, the function `Ano_CrossValid()` automatically computes a 
climatology for each starting date without taking that starting date and 
subtracts it to the original data:
```r
ano <- Ano_CrossValid(data$mod, data$obs, memb = FALSE)
```

The anomalies can be plotted with `PlotAno()`:
```r
PlotAno(ano$ano_exp, ano$ano_obs, selected_sdates,
        toptitle = paste0(c("Experiment A", "Experiment B"),
                          ": cross validation 'tas' anomalies over Atlantic region."),
        ytitle = c("K", "K"), linezero = TRUE,
        fileout = paste0("stat_ano_exp", c("A", "B"), "_obsX.eps"))
```
<img src="stat_ano_expA_obsX.png" width="600" />
<img src="stat_ano_expB_obsX.png" width="600" /> 

To fulfill the bias correction of the forecasts, i.e. transforming the forecast
data from the biased model mean state to the real observed mean state, the 
climatology of the observation has to be added to the anomalies of the 
forecasts. Since the score functions take anomalies as working units this step 
is not needed for verification.

Describing fields<a name="describing"></a>
-----------------
While the drift correction had effect on series along the forecast time, the 
description and filtering functions are aimed to manipulate data also along 
the actual time.  
The functions to describe fields allow you to compute the trend of a field,
its regression with another field, its frequency spectrum and its 
autocorrelation.

### Linear trend and regression
`Trend()` fits first degree linear models by least square fitting, taking as
dependent variable a field given to its argument `var` and a line of slope 1 
as independent variable. A linear model is fitted for each actual time-series
of each dataset, member, lead-time, latitude and longitude in `var` and the
coefficient, intercept, confidence interval and detrended data are provided as 
output. The confidence intervals rely on a T-student distribution.
  
`Trend()` can be useful, for example, in global warming scenarios, frequent 
in seasonal and decadal prediction, to detect, describe and remove any 
constantly increasing or decreasing trend. The following example computes the 
trends of all the datasets at each lead-time and plots the slope of the trends
of the experimental datasets with `PlotVsLTime()` (see 
[**Visualisation**](visualisation.md)):
```r
trend_exp <- Trend(Mean1Dim(ano$ano_exp, 2))
trend_obs <- Trend(Mean1Dim(ano$ano_obs, 2))
PlotVsLTime(trend_exp$trend, 
            toptitle = "Slopes of trends of 'tas' over Atlantic region", 
            ytitle = "K/year",
            monini = 12, freq = 1, 
            listexp = c('Experiment A', 'Experiment B'),
            fileout = 'stat_trend_expA_expB.eps')
```
<img src="stat_trend_expA_expB.png" width="600" />

In this case, the slopes of the trends are nearly zero at all lead-times. The 
raw anomalies of the experiment A and observations are plotted next, side to 
side with the detrended ones:
```r
PlotAno(plyr::take(ano$ano_exp, 1, 1), ano$ano_obs, 
        selected_sdates,
        toptitle = "Experiment A: c-v 'tas' anomalies over Atlantic region.",
        ytitle = "K", linezero = TRUE,
        fileout = paste0("stat_ano_expA_obsX.eps"))
PlotAno(InsertDim(plyr::take(trend_exp$detrended, 1, 1), 2, 1), 
        InsertDim(trend_obs$detrended, 2, 1), 
        selected_sdates,
        toptitle = "Experiment A: detrended c-v 'tas' anomalies over Atlantic region.",
        ytitle = "K", linezero = TRUE,
        fileout = paste0("stat_detr_ano_expA_obsX.eps"))
```
<img src="stat_ano_expA_obsX.png" width="600" />
<img src="stat_detr_ano_expA_obsX.png" width="600" />

The parameter `posTR` of `Clim()` allows to specify the position in `var` of 
the actual time dimension, which by default is expected to be in second place 
as it is also expected that the input field has been averaged along members 
with `Mean1Dim()`. It also allows to compute the trend along other dimensions 
than actual time; for example, forecast time.  
The parameter `interval` allows to specify the number of time steps between
consecutive values along the `posTR`th dimension, `1` by default.

The function `Consist_Trend()` only takes into account those lead-times and 
starting dates at which observations are available (when using on 
2-dimensional fields, a lead-time or starting date will only be discarded if 
all its grid values are not available).

`Regression()`, in contrast to `Trend()`, takes a second field as parameter 
and uses it as independent variable to fit the linear model. It can be useful 
to detect, describe and filter any relationship between two fields or to
quantify the  
As a matter of an example, let's suppose the experiment A is under the effect 
of the phenomena X and Y but only the phenomenon Y is relevant for the study 
case. It may be threfore needed to remove any variability of A related to X. 
Let's suppose that the experiment B is only under the effect of the phenomena
X. Then the variability of A related to Y can be isolated by subtracting to 
it its regression with B:
```r
ano_expA_Y <- Regression(plyr::take(ano$ano_exp, 1, 1),
                         plyr::take(ano$ano_exp, 1, 2))
PlotAno(ano_expA_Y$filtered, ano$ano_obs,
        selected_sdates,
        toptitle = "Experiment A: c-v 'tas' anomalies over Atlantic region, X subtracted.",
        ytitle = "K", linezero = TRUE,
        fileout = paste0("stat_ano_expA_Y_obsX.eps"))
```
<img src="stat_ano_expA_Y_obsX.png" width="600" />

### Frequency spectrum


### Autocorrelation
Knowing whether a field is time-correlated with itself is a crucial step to 
determine whether the size of the sample is large enough to yield a robust 
validation. It is also useful to build synthetic models with the same time 
correlation behaviour as another.  
The function `Alpha()` serves to compute the best estimate of the 
autocorrelation of a signal at 1 time-step lag. Knowing the best estimate 
is needed to __________.  

`Alpha()` has two parameters to enable linear detrending and/or filtering of 
frequency peaks, convenient if ___________.

Assuming we have followed the same data retrieval process as in the 
[**Example**](example.md), we can calculate the best estimate of 
autocorrelation at lag 1 of the bias-corrected, detrended, observed anomalies 
as follows:
```r
clim <- Clim(data$mod, data$exp)
ano <- Ano(clim$clim_exp, clim$clim_obs)
autocorr <- Alpha(ano$obs[1, 1, , 1], detrend = TRUE)
```
Internally, `Alpha()` calls `FitAcfCoef()`. `FitAcfCoef()` computes the 
autocorrelation from the first estimates of the autocorrelation at lags 1 and 
2, that can be obtained providing the original time series to the R built-in 
function `acf()`:
```r
estacf <- acf(ano$obs[1, 1, , 1], plot = FALSE)$acf
autocorr <- FitAcfCoef(max(estacf[2], 0), max(estacf[3], 0))
```  
`FitAutocor()` estimates the autocorrelation from the first autocorrelation 
estimate and a window where to seek in with a dichotomial method.
```r
estacf <- acf(ano$obs[1, 1, , 1], plot = FALSE)$acf
autocorr <-  FitAutocor(estacf, c(-1, 1), 0.01)

Filtering fields<a name="filtering"></a>
----------------
Depending on the source and nature of the field to study and on the study 
case it is often needed to isolate, remove or modulate its components. 

### Detrending data



### Frequency filtering
Smoothing, Spectrum, Filter

```

Generating derivative fields<a name="generating_der"></a>
----------------------------

Seasonal means
Season
Should this go to data retrieval?

Cathegorizing data
ProbBins

Statistical downscaling
StatSeasAtlHurr

Generating new fields<a name="generating_new"></a>
---------------------

Empirical models
GenSeries, Toymodel, Eno, EnoNew

