---
output:
  pdf_document: default
  html_document: default
---
Statistics
==========
The **Statistics** module consists of functions that are commonly used in the 
forecast verification process to modify, describe or generate **fields**, a 
field being a series of modelled or observed measurements for a certain physic 
variable of interest.

These functions can be classified in 5 groups as a function of their purpose:

  - [**Normalizing fields**](#normalizing): `Clim()`, `Ano()` and 
`Ano_Crossvalid()`.
  - [**Describing fields**](#describing): `Trend()`, `Consist_Trend()`, 
`Regression()`, `FitAcfCoef()`, `Alpha()`, `FitAutocor()`, `Spectrum()`, 
`Eno()` and `EnoNew()`.
  - [**Filtering fields**](#filtering): `Trend()`, `Consist_Trend()`, 
`Regression()`, `Smoothing()` and `Filter()`.
  - [**Generating derivative fields**](#generating_der): `Season()`, 
`StatSeasAtlHurr()` and `ProbBins()`.
  - [**Generating synthetic fields**](#generating_synth): `GenSeries()` and 
`Toymodel()`.

It is important, to master these functions, to have in mind the concepts 
introduced in **Data retrieval** as well as the structure and meaning of the 
dimensions of the data arrays provided by the **Data retrieval** module used 
throughout in s2dverification:
```r
  c(n. of datasets, n. of members, n. of starting dates, n. of lead-times, n. of latitudes, n. of longitudes)
```
the last two being present in function of the parameter `output`. Review 
[**Data retrieval**](data_retrieval.md) to settle these concepts. 

The explanations in the following sections will be based on the same data used 
in **Data retrieval**. See [**Data retrieval**](data_retrieval.md) for details.
```r
library(s2dverification)
expA <- list(name = 'experimentA',
             path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                              '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc'))
expB <- list(name = 'experimentB',
             path = file.path('/path/to/experiments/$EXP_NAME$/monthly_mean',
                              '$VAR_NAME$/$VAR_NAME$_$START_DATE$.nc'))
obsX <- list(name = 'observationX',
             path = file.path('/path/to/observations/$OBS_NAME$/monthly_mean',
                              '$VAR_NAME$/$VAR_NAME$_$YEAR$$MONTH$.nc'))
sdates <- paste0(1991:2000, '1101')
data <- Load('tas', list(expA, expB), list(obsX),
             sdates = sdates,
             leadtimemin = 2, leadtimemax = 13,
             latmin = -10, latmax = 60,
             lonmin = 100, lonmax = 250,
             output = 'areave', grid = 't106grid',
             method = 'distance-weighted')
```
The dimensions of the data are:
```r
print(dim(data$mod))
# [1] 2, 5, 10, 12
print(dim(data$obs))
# [1] 1, 1, 10, 12
```

Normalizing fields<a name="normalizing"></a>
------------------
Due to systematic errors in the models their mean behaviour along the forecast
time can differ or depart from the mean behaviour of the observed data over 
that forecast time period, a phenomenon also known as **drift**.  
It is hence a standard practice in climate forecast verification to compare 
only the deviations of the experimental and observational data from their mean 
behaviour along the forecast time. For this reason most of the functions in 
s2dverification use series of deviations as inputs and provide series of 
deviations as outputs.

The mean behaviour along the forecast time of a data set, also known as the 
**climatology**, can be computed with multiple methods. The deviation from 
the climatology is known as the **anomaly** or, more explicitly, 
drift-corrected anomaly, and it can be calculated by simply subtracting the 
climatology of a data set to its raw data.

A general purpose climatology could be computed as follows (the 3rd dimension 
corresponds to the starting dates or model runs):
```r
clim_exp <- apply(data$mod, 1:length(dim(data$mod))[-3], mean, na.rm = TRUE)
```
or with the function `Mean1Dim()` in s2dverification:
```r
clim_exp <- Mean1Dim(data$mod, 3)
```

s2dverification, however, builds upon the per-pair method to compute 
climatologies, which is implemented in the function `Clim()`.  
The `Clim()` parameters `kharin` and `NDV` allow to select two other more 
refined methods: the trend method and the initial-conditions method. All of 
them are documented in
```
Fučkar N, Volpi D, Guemas V, Doblas-Reyes F, 2014, A posteriori adjustment of near-term climate predictions: Accounting for the drift dependence on the initial conditions. Geophysical Research Letters, 41 (14), 5200–5207, doi:10.1002/2014GL060815
```
Following the per-pair method the climatology of an experimental dataset is 
obtained, as in the general purpose climatology, by calculating an average 
across the starting dates at each forecast time, yielding a time series 
defined along the forecast times of the experiment. The climatology of an 
observational dataset is obtained by computing, for each forecast time, the 
average of the observations that date-correspond that leadtime of each 
starting date.  
To make easy this process `Load()` dispones both experimental and 
observational data in a forecast-like arrangement.

`Clim()` and the per-pair method only account, at each forecast time, for the 
starting dates for which data is available in all the experimental and 
observational datasets, hence both experimental and observational data must be
provided as inputs:

```r
clim <- Clim(data$mod, data$obs)
```

The raw data and the climatologies can be visualized with `PlotAno()` and 
`PlotClim()`, respectively:
```r
selected_sdates <- gsub('1101', '1201', sdates)
PlotAno(data$mod, data$obs, selected_sdates, 
        toptitle = paste0(c("Experiment A", "Experiment B"), 
                          ": Raw 'tas' over North Pacific region."), 
        ytitle = c("K", "K"), 
        fileout = paste0("stat_raw_exp", c("A", "B"), "_obsX.eps")) 
PlotClim(clim$clim_exp, clim$clim_obs, monini = 12,
         toptitle = "Per-pair 'tas' climatologies over the North Pacific region.", 
         ytitle = "K",
         listexp = c('Experiment A', 'Experiment B'),
         listobs = c('Observation X'),
         fileout = "stat_clim_expA_expB_obsX.eps")
```
<img src="stat_raw_expA_obsX.png" width="600" />
<img src="stat_raw_expB_obsX.png" width="600" /> 

Each coloured curve in the `PlotAno()` figures corresponds to a starting date, 
with the various ensemble members and the ensemble mean in bold. The coloured
area is delimited by the minimum and maximum ensemble values. 

<img src="stat_clim_expA_expB_obsX.png" width="600" /> 

Each plot in the `PlotClim()` figure corresponds to the climatology of a 
member of the corresponding experiment or observation. 

See [**Visualisation**](visualisation.md) for a complete explanation of the 
plotting functions.

Usually, if working with atmospheric variables, it is not relevant when 
computing climatologies to make a distinction among members because they are 
considered to be equivalent. It is possible, after seeing they are indeed 
equivalent, to compute the ensemble mean climatology with
```r
exp_ens_clim <- Mean1Dim(clim$clim_exp, 2)
obs_ens_clim <- Mean1Dim(clim$clim_obs, 2)
```
or, if sure a priori of their equivalence, by setting the parameter 
`memb = FALSE` in `Clim()`: 
```r
clim <- Clim(data$mod, data$obs, memb = FALSE)
```

If working with oceanic variables this may not necessarly apply. See
```
Du, H., F.J. Doblas-Reyes, J. García-Serrano, V. Guemas, Y. Soufflet and B. Wouters (2012). Sensitivity of decadal predictions to the initial atmospheric and oceanic perturbations. Climate Dynamics, 39, 2013-2023, doi:10.1007/s00382-011-1285-9. http://link.springer.com/article/10.1007%2Fs00382-011-1285-9 Fig 2d is a very good example. If only one climatology is used for this variable, the skill is substantially lower.
```

Once the climatologies are computed, the anomalies can be obtained with 
`Ano()`, that subtracts the climatologies to each member, starting date,
latitude and longitude of their corresponding dataset:
```r  
ano_exp <- Ano(data$mod, clim$clim_exp)
ano_obs <- Ano(data$obs, clim$clim_obs)
```

All the starting dates of an experiment contribute to the per-pair climatology 
and, when computing the anomaly of that starting date, its contribution to the 
climatology is subtracted, which can imply a loss of information [citation]. 
To avoid this, the function `Ano_CrossValid()` automatically computes a 
climatology for each starting date without taking that starting date into 
account and subtracts it to the original data:
```r
ano <- Ano_CrossValid(data$mod, data$obs, memb = FALSE)
```

The anomalies can be plotted with `PlotAno()`:
```r
PlotAno(ano$ano_exp, ano$ano_obs, selected_sdates,
        toptitle = paste0(c("Experiment A", "Experiment B"),
                          ": c.v. 'tas' anomalies over North Pacific region."),
        ytitle = c("K", "K"), linezero = TRUE,
        fileout = paste0("stat_ano_exp", c("A", "B"), "_obsX.eps"))
```
<img src="stat_ano_expA_obsX.png" width="600" />
<img src="stat_ano_expB_obsX.png" width="600" /> 

To fulfill the bias correction of the forecasts, i.e. transforming the forecast
data from the biased model mean state to the real observed mean state, the 
climatology of the observation has to be added to the anomalies of the 
forecasts. Since the score functions take anomalies as working units this step 
is not needed for verification.

Describing fields<a name="describing"></a>
-----------------
While the drift correction had effect on series along the forecast time, the 
description and filtering functions are aimed to analyse and manipulate data 
also along the actual time (or starting dates).  
The functions to describe fields allow you to compute the trend of a field,
its regression with another field, its frequency spectrum and its 
autocorrelation.

### Linear trend and regression
`Trend()` fits first degree linear models by least square fitting taking as
dependent variable a field given to its argument `var` and a line of slope 1 
as independent variable. A linear model is fitted for each actual time-series
of each dataset, member, lead-time, latitude and longitude in `var`. The
coefficient and intercept of the linear model, the confidence interval relying 
on a T-student distribution and the detrended data are provided as output.
  
`Trend()` can be useful, for example, in global warming scenarios (frequent 
in seasonal and decadal prediction) to detect, describe and remove any 
constantly increasing or decreasing trend. The following example computes the 
trends of all the datasets at each lead-time and plots the slope of the trends
of the experimental datasets with `PlotVsLTime()` (see 
[**Visualisation**](visualisation.md)):
```r
trend_exp <- Trend(Mean1Dim(ano$ano_exp, 2))
trend_obs <- Trend(Mean1Dim(ano$ano_obs, 2))
PlotVsLTime(trend_exp$trend, 
            toptitle = "Slopes of trends of 'tas' over North Pacific region", 
            ytitle = "K/year",
            monini = 12, freq = 1, 
            listexp = c('Experiment A', 'Experiment B'),
            fileout = 'stat_trend_expA_expB.eps')
```
<img src="stat_trend_expA_expB.png" width="600" />

In this case the slopes of the trends are nearly zero at all lead-times. The 
raw anomalies of the experiment A and observations are plotted next, side to 
side with the detrended ones:
```r
PlotAno(plyr::take(ano$ano_exp, 1, 1), ano$ano_obs, 
        selected_sdates,
        toptitle = "Experiment A: c.v. 'tas' anomalies over North Pacific region.",
        ytitle = "K", linezero = TRUE,
        fileout = paste0("stat_ano_expA_obsX.eps"))
## PlotAno() requires that the members dimension is present in its inputs. 
## Since we squeezed the members dimension before computing the trend, we need
## to put it again with InsertDim(): a dimension of length 1 at the 2nd place.
PlotAno(InsertDim(plyr::take(trend_exp$detrended, 1, 1), 2, 1), 
        InsertDim(trend_obs$detrended, 2, 1), 
        selected_sdates,
        toptitle = "Experiment A: detrended c.v. 'tas' anomalies over North Pacific region.",
        ytitle = "K", linezero = TRUE,
        fileout = paste0("stat_detr_ano_expA_obsX.eps"))
```
<img src="stat_ano_expA_obsX.png" width="600" />
<img src="stat_detr_ano_expA_obsX.png" width="600" />

Since the anomaly members have been averaged to compute the trend, the provided
detrended data by `Trend()` is also an ensemble average.

The parameter `posTR` of `Trend()` allows to specify the position in `var` of 
the actual time dimension, which by default is expected to be in second place 
as it is also expected that the input field has been averaged along members 
with `Mean1Dim()`. It also allows to compute the trend along other dimensions 
than actual time; for example, forecast time.  
The parameter `interval` allows to specify the number of time steps between
consecutive values along the `posTR`th dimension, `1` by default.

The function `Consist_Trend()` only takes into account those lead-times and 
starting dates at which observations are available (when using on 
2-dimensional fields, a lead-time or starting date will only be discarded if 
all its grid values are not available).

`Regression()`, in contrast to `Trend()`, takes a second field as parameter 
and uses it as independent variable to fit the linear model. It can be useful 
to detect, describe and filter any relationship between two fields or to
quantify the  
As example, let's suppose the experiment A is under the effect of the 
phenomena X and Y but only the phenomenon Y is relevant for the study case. It 
may be threfore needed to remove any variability of A related to X. Let's 
suppose that the experiment B is only under the effect of the phenomena X. 
Then the variability of A related to Y can be isolated by subtracting its 
regression with B:
```r
ano_expA_Y <- Regression(plyr::take(ano$ano_exp, 1, 1),
                         plyr::take(ano$ano_exp, 1, 2), posREG = 3)
PlotAno(ano_expA_Y$filtered, ano$ano_obs,
        selected_sdates,
        toptitle = "Experiment A: c.v. 'tas' anomalies over North Pacific region, X subtracted.",
        ytitle = "K", linezero = TRUE,
        fileout = "stat_ano_expA_Y_obsX.eps")
```

### Frequency spectrum
The function `Spectrum()` estimates the frequency components of the input 
time series relying on the R base `spectrum()` function. Additionally to the
frequency and power of each component it also computes the 95% and 99% 
significance levels _____for accepting each frequency as being a component of the
provided series_____. These are estimated with a Monte-Carlo method.

The spectrum of the ensemble mean of the first starting date of the experiment
A is computed next and the frequency, power and significance levels of the 
most powerful component are printed:
```r
components <- Spectrum(Mean1Dim(plyr::take(ano$ano_exp, c(1, 3), c(1, 1)), 2))
print(components[1, ])
# [1] 0.083333333 0.001797855 0.035540044 0.043555352
```

`Spectrum()` assumes the input data is evenly spaced in time.

Check [**Filtering fields**](filtering.md) for an example of filtering 
frequencies estimated with `Spectrum()`. 

### Autocorrelation and persistence
Knowing whether a field is time-correlated with itself is a crucial step to 
determine whether the size of the sample is large enough to yield a robust 
validation. It is also useful to build synthetic models with the same time 
correlation behaviour as another. Knowing the best estimate of autocorrelation
at lag 1 is needed to _________.

`FitAutocor()` estimates the autocorrelation from the first autocorrelation 
estimate, that can be obtained with the R built-in function `acf()`, a 
window where to seek in with a dichotomial method and a precision factor:
```r
estacf <- acf(ano$ano_obs[1, 1, , 1], plot = FALSE)$acf
autocorr <-  FitAutocor(estacf, c(-1, 1), 0.01)
print(autocorr)
# [1] 0
```

The obtained autocorrelation of 0 means that _____________.

`FitAcfCoef()` computes _____the best estimate of the 
persistence/autocorrelation at lag 1 from the first estimates of the 
autocorrelation at lags 1 and 2, that can be 
obtained providing the original time series to the R built-in function `acf()`:
```r
estacf <- acf(ano$ano_obs[1, 1, , 1], plot = FALSE)$acf
persistence <- FitAcfCoef(max(estacf[2], 0), max(estacf[3], 0))
```  

The function `Alpha()` automatically computes the persistence by calling 
`FitAcfCoef()` after computing the first estimates with `acf()`:
```r
persistence <- Alpha(ano$ano_obs[1, 1, , 1])
print(persistence)
# [1] 0
```

`Alpha()`, additionally, has two parameters to enable linear detrending and/or 
filtering of frequency peaks prior to the computation of the persistence, 
convenient if ___________.

### Effective sample size
`Eno()` and `EnoNew()` compute the effective number of independent data.  
While `Eno()` relies on the `eno` function from "rclim.txt" from Caio Coelho 
and operates on s2dverification common formatted arrays, `EnoNew()` relies on
the method described in
```
Guemas V., Auger L., Doblas-Reyes F., JAMC, 2013
```
and operates on time series and applies, if specified in the parameters `detrend` 
and `filter`, a detrending and filtering of frequency peaks, similarly to
`Alpha()`.
```r
eno <- Eno(data$mod, 3)
print(unique(as.vector(eno)))
# [1] 10.000000  9.104402  6.805723  8.270278  9.492232  8.783534  7.756835
# [8]  9.270968  8.025609

```
This means that __________.


Filtering fields<a name="filtering"></a>
----------------
Depending on the source and nature of the field to study and on the study 
case it is often needed to isolate, remove or modulate its components. 

As seen in [**Describing fields**](#describing), the functions `Trend()` and
`Consist_Trend()` remove linear trends and provide detrended data in the 
component `detrended` of their output. Likewise, `Regression()` removes 
behaviours related to other fields and provides the filtered data in the
output component `filtered`.

### Running means

`Smoothing()` calculates running means to remove high frequency variability. 
The parameter `runemanlen` allows to specify the running window width and 
`numdimt` which dimension to smooth along (4 by default; along forecast time). 
In this __not_very_useful___ example the anomalies are smoothed along the 
actual time with a window of 3 months to remove the inter-seasonal variability. 
The first and last forecast months are lost as appreciated in the figure:
```r
smoothed_ano_exp <- Smoothing(ano$ano_exp, 3)
smoothed_ano_obs <- Smoothing(ano$ano_obs, 3)
PlotAno(plyr::take(smoothed_ano_exp, 1, 1), smoothed_ano_obs,
        selected_sdates,
        toptitle = "Experiment A: smoothed c.v. 'tas' anomalies over North Pacific region.",
        ytitle = "K", linezero = TRUE,
        fileout = "stat_smooth_ano_expA_obsX.eps")
```
<img src="stat_smooth_ano_expA_obsX.png" width="600" />

### Frequency filtering
`Filter()` filters a specified frequency from the input data. The filtering is 
performed by dichotomy, seeking for the frequency around the specified one and 
for the phase that maximizes the signal to subtract to the data.  
The maximization of the signal to subtract relies on a minimization of the 
mean square differences between the data and a cosine of each frequency and 
phase within the seek range.

The following example filters all the frequencies in experiment A ___with a 
power beyond the 99% signification level___ with the purpose of _____________:
```r
ens_mean_ano_expA <- Mean1Dim(plyr::take(ano$ano_exp, 1, 1), 2)
for (sdate in 1:length(sdates)) {
  spectrum <- Spectrum(ens_mean_ano_expA[1, sdate, ])
  for (freq in 1:dim(spectrum)[1]) {
    if (spectrum[freq, 2] > spectrum[freq, 4]) {
      ens_mean_ano_expA[1, sdate, ] <- Filter(ens_mean_ano_expA[1, sdate, ], 
                                              spectrum[freq, 1])
    }
  }
}
PlotAno(InsertDim(ens_mean_ano_expA, 2, 1), ano$ano_obs,
        selected_sdates, 
        toptitle = "Experiment A: filtered c.v. 'tas' anomalies over North Pacific region.",
        ytitle = "K", linezero = TRUE,
        fileout = "stat_filter_ano_expA.eps")
```
<img src="stat_filter_ano_expA.png" width="600" />

Generating derivative fields<a name="generating_der"></a>
----------------------------
Some of the functions in the package generate fields based on data from other 
fields, for example seasonal means from monthly time series or downscaled data.

### Seasonal means
`Season()` computes seasonal means from data objects in the common 
s2dverification structure along the forecast time dimension, which is expected
to be, by default, the 4th. The data is expected to be evenly spaced along
that dimension.  
The initial month of the time-series must be specified, as well as the initial
and final months of the desired season. The parameter `posdim` allows to 
specify the dimension to compute the seasonal mean along, 4 by default, 
forecast time.

The following example loads the data without area averaging, i.e. gridded data, 
and computes and plots the MAM seasonal mean climatology of each grid cell over 
the North Pacific region:
```r
data_map <- Load('tas', list(expA, expB), list(obsX),
                 sdates = sdates,
                 leadtimemin = 2, leadtimemax = 13,
                 latmin = -10, latmax = 60,
                 lonmin = 100, lonmax = 250,
                 output = 'lonlat', grid = 't106grid',
                 method = 'distance-weighted')
clim_map <- Clim(data_map$mod, data_map$obs, memb = FALSE)
mam_clim_exp <- Season(clim_map$clim_exp, posdim = 2, 
                       monini = 12, moninf = 3, monsup = 5)
mam_clim_obs <- Season(clim_map$clim_obs, posdim = 2, 
                       monini = 12, moninf = 3, monsup = 5)
brks <- round(seq(min(mam_clim_exp, mam_clim_obs, na.rm = TRUE), 
                  max(mam_clim_exp, mam_clim_obs, na.rm = TRUE), 
                  length.out = 51), 2)
cols <- clim.colors(50)
png('stat_season_mam_expA.png', width = 800, height = 600, type = "Xlib")
PlotEquiMap(mam_clim_exp[1, 1, , ], data_map$lon, data_map$lat, 
            toptitle = "Experiment A: Spring (MAM) 'tas' climatologies.",
            units = "K", brks = brks, cols = cols, subsampleg = 10)
dev.off()
png('stat_season_mam_obsX.png', width = 800, height = 600, type = "Xlib")
PlotEquiMap(mam_clim_obs[1, 1, , ], data_map$lon, data_map$lat, 
            toptitle = "Observation X: Spring (MAM) 'tas' climatologies.",
            units = "K", brks = brks, cols = cols, subsampleg = 10)
dev.off()
```
<img src="stat_season_mam_expA.png" width="600" />
<img src="stat_season_mam_obsX.png" width="600" />

### Cathegorizing data
`ProbBins()`

### Statistical downscaling
`StatSeasAtlHurr()`

Generating synthetic fields<a name="generating_synth"></a>
---------------------------
To perform statistical/inference tests it is required to generate new data 
obtained with empirical models that comply with a given set of distribution 
parameters. 

`GenSeries()` generates a series with a specified length, autocorrelation at 
lag 1, mean and standard deviation. For example, a synthetic model can be 
parametrized to generate a forecast with the same parameters as the 
experiment A:
```r
synth_exp_A <- apply(plyr::take(data$mod, 1, 1), c(1, 2, 3),
                     function(x) {
                       GenSeries(length(x), Alpha(x), mean(x, na.rm = TRUE),
                                 sd(x, na.rm = TRUE))
                     })                    
```

`Toymodel()` directly generates forecasts in the s2dverification common data 
structure, imitating seasonal to decadal forecasts. The toymodel is based on 
the model presented in 
```
Weigel et al. (2008) QJRS
```
with an extension to consider non-stationary distributions prescribing a 
linear trend. The toymodel allows to generate an aritifical forecast based on 
real obsevations provided by the input (from `Load()`) or artificially 
generated observations based on the input parameters `sig` and `trend`. The 
forecast can be specfied for any number of starting dates, lead-times and 
ensemble members. It imitates components of a forecast: (1) predictabiltiy (2) 
forecast error (3) non-stationarity and (4) ensemble generation.

The following example generates a forecast for seasonal prediction with 
synthetic observations, and plots its anomalies:
```r
nstartd <- 30
sdates_toy <- paste0(1981:(1981 + nstartd), '1101')
toy_forecast <- Toymodel(alpha = 0.1, beta = 0.3, gamma = 1, sig = 1, 
                         trend = 0.02, nstartd = nstartd, nleadt = 12, 
                         nmemb = 10)
ano_toy <- Ano_CrossValid(toy_forecast$mod, toy_forecast$obs)
PlotAno(ano_toy$ano_exp, ano_toy$ano_obs, sdates_toy,
        toptitle = "Toy forecast: anomalies.",
        ytitle = "units", linezero = TRUE,
        fileout = "stat_toy_forecast_ano.eps")
```
<img src="stat_toy_forecast_ano.png" width="600" />

It is possible, however, to generate model data from observational data from
`Load()`. The only required parameters are, then, the predictability, error 
standard deviation and factor on linear trend to sample uncertainty:

```r
toy_forecast_C <- Toymodel(alpha = 0.1, beta = 0.3, gamma = 1, nmemb = 10, 
                           obsini = data$obs)
PlotAno(toy_forecast_C$mod, toy_forecast_C$obs, selected_sdates,
        toptitle = "Toy forecast from observation X: anomalies.",
        ytitle = "K", linezero = TRUE,
        fileout = "stat_toy_expC_obsX_ano.eps")
```
