---
author: "Nuria Perez"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteEngine{knitr::knitr}
  %\VignetteIndexEntry{Data Storage and Retrieval}
  %\usepackage[utf8]{inputenc}
---

Data Storage and Retrieval
-----------------------------------------

CSTools is aim for post-processing seasonal climate forecast with state-of-the-art methods. However, a necessary step prior to the this goes from downloading existing datasets to loaded in the R session. Therefore, some recommendations and key points to take into account are gathered here in order to facilitate the use of CSTools. 
 
### 1. System requirements

The first question may come to a new user is the requirements of my computer to run CSTools. Here, the list of most frequent needs:
- netcdf library version 4.1 or later
- cdo (I am currently using 1.6.3)
- R 3.4.2 or later

On the other hand, the computational power of a computer could be a limitation, but it will depends on the size of the data that the users need for their analysis. For instance, they can estimate the memory they will require by multiplying the following values:

- Area of the study region (km^2)
- Area of the desired grid cell (km) (or square of the grid cell size)
- Number of models + 1 observational dataset
- Forecast time length (days or months)
- Temporal resolution (1 for daily, 4 for 6 hourly or 24 for daily data)
- Hindcast length (years)
- Number of start date (or season)
- Number of members
- Extra factor for functions computation(*)

For example, if they want to use the hindcast of 3 different seasonal simulations with 9 members, in daily resolution, for performing a regional study let's say in a region of 40000 km2 with a resolution of 5 km:

> 200km x 200km / (5km * 5km) * (3 + 1) models * 214 days * 30 hindcast years * 9 members x 2 start dates x 8 bytes ~ 6 GB

(*)Furthermore, some of the functions need to duplicated or triplicate the inputs for performing their analysis. Therefore, between 12 and 18 GB of RAM memory would be necessary, in this example.


### 2. Data storage recommendations

CSTools package includes a function to read the files and have the data available in the R session memory to conduct the post-processing. There are alternative, for instance, they can use another tool to read the data from files and then convert it to the class 's2dv_cube' with s2dv.cube function. They can use startR or ndcf4 packages. If they keep facing problems to convert the data to that class, they can just skip it and work with the functions without the prefix 'CST_'. In this case, they will be able to work with the basic class 'array'. 

Summarizing, the loading step could be done:
- using CST_Load function
- using other tools the original Load() from s2dverification, startR package or the basic ncd4
   + conver to class s2dv_cube with s2dv.cube() or
   + directly, using the functions without the 'CST_' prefix which work on arrays.

!!!!!add links to documentation


### 3. CST_Load example



```
library(CSTools)
library(zeallot)
path <- "/esarchive/exp/meteofrance/system6c3s/$STORE_FREQ$_mean/$VAR_NAME$_f6h/$VAR_NAME$_$START_DATE$.nc"
ini <- 1993
fin <- 2012
month <- '05'
start <- as.Date(paste(ini, month, "01", sep = ""), "%Y%m%d")
end <- as.Date(paste(fin, month, "01", sep = ""), "%Y%m%d")
dateseq <- format(seq(start, end, by = "year"), "%Y%m%d")
c(exp, obs) %<-%  CST_Load(var = 'sfcWind',
                           exp = list(list(name = 'meteofrance/system6c3s', path = path)),
                           obs = 'erainterim',
                           sdates = dateseq, leadtimemin = 2, leadtimemax = 4,
                           lonmin = -19, lonmax = 60.5, latmin = 0, latmax = 79.5,
                           storefreq = "daily", sampleperiod = 1,  nmember = 9,
                           output = "lonlat", method = "bilinear",
                           grid = "r360x180")
```


Extra lines to see the size of the objects and visualize the data:

```
library(pryr)
object_size(exp)
# 27.7 MB
object_size(obs)
# 3.09 MB
library(s2dv)
PlotEquiMap(exp$data[1,1,1,1,,], lon = exp$lon, lat= exp$lat,
            filled.continents = FALSE, fileout = "Meteofrance_r360x180.png")
```


![Meteofrance](../vignettes/Figures/Meteofrance_r360x180.png)
