---
title: "Analogs based on large scale for downscaling"
author: "M. Carmen Alvarez-Castro (CMCC, Italy)"
date: "November 2020"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteEngine{knitr::knitr}
  %\VignetteIndexEntry{Analogs}
  %\usepackage[utf8]{inputenc}
---

<!--
```{r, echo = FALSE}
knitr::opts_chunk$set(eval = FALSE)
```
-->
## Downscaling seasonal forecast data using Analogs

In this example, the seasonal temperature forecasts, initialized in october, will be used to perform a downscaling in the Balearic Islands temperature using the cmcc system 3 seasonal forecasting system from the Euro-Mediterranean Center of Climate Change (CMCC), by computing Analogs in Sea level pressure data (SLP) in a larger region (North Atlantic). The first step will be to load the data we want to downscale (i.e. cmcc) in the large region (i.e North Atlantic) for temperature (predictand) and SLP (predictor) and same variables and region for a higher resolution data (ERA5). In a second step we will interpolate the model to the resolution of ERA5. In a third step we will find the analogs using one of the three criterias. In a four step we will get the downscaled dataset in the region selected (local scale, in this case Balearic Islands)

## 1. Introduction of the function

For instance if we want to perform a temperature donwscaling in Balearic Island for October we will get a daily series of temperature with 1 analog per day, the best analog. How we define the best analog for a certain day? This function offers three options for that:

(1) The day with the minimum Euclidean distance in a large scale field: using i.e. pressure or geopotencial height as variables and North Atlantic region as large scale region.  The Atmospheric circulation pattern in the North Atlantic (LargeScale) has an important role in the climate in Spain (LocalScale). The function will find a day with the most similar pattern in atmospheric circulation in the database (obs, slp in ERA5) to the day of interest (exp,slp in model). Once the date of the best analog is found, the function takes the associated temperature to that day (obsVar, tas in ERA5), with a subset of the region of interest (Balearic Island)

(2) Same that (1) but in this case we will search for analogs in the local scale (Balearic Island) instead o large scale (North Atlantic. Once the date of the best analog is found, the function takes the associated temperature to that day (obsVar, t2m in ERA5), with a subset of the region of interest (Balearic Island)

(3) Same that (2) but here we will serach for analogs with higher correlation at local scale (Balearic Island) and instead of using SLP we will use t2m. 


In particular the _Analogs Method_  use a nonlinear approach that follows (**Analogs**; Yiou et al. 2013) 

An efficient implementation of Analogs is provided for CSTools by the `CST_Analogs()` function. 

### 1. Preliminary setup

In order to run the examples in this vignette, the *CSTools* package and some other support R packages need to be loaded by running:

```{r}
install.packages('CSTools')
library(CSTools)
library(multiApply)
library(s2dverification)
library(fields)
```
### 2. Load data 
Here we show to options for the example, a) loading data using CST_Load, and b) loading the experimental data included in CSTools package (lonlat_data)

### 2.a- Load data using CST_Load
The parameters defined are the initializing month and the variables:


```{r cars}
mth = '10'
temp = 'tas'
slp = 'psl'

require(zeallot)
```


The simulations available for this model cover the period 1993-2016. For ERA 5 from 1979 to the present days. For this example we will just use from 2000 to 2006, So, the starting and ending dates can be defined by running the following lines:


```r
ini <- 2000
fin <- 2006
start <- as.Date(paste(ini, mth, "01", sep = ""), "%Y%m%d")
end <- as.Date(paste(fin, mth, "01", sep = ""), "%Y%m%d")
dateseq <- format(seq(start, end, by = "year"), "%Y%m%d")
``` 


The grid in which all data will be interpolated for the downscaling  should be also specified. The observational dataset used in this example is the Era5. 


```
r grid <- '1440x721'
obs <- "era5"
```

Using the `CST_Load` function from **CSTool package**, the data available in our data store can be loaded. The following lines show how this function can be used. Here, the data is loaded from a previous saved `.RData` file:
Ask carmen.alvarez-castro at cmcc.it or nuria.perez at bsc.es for the data to run the recipe.

```r
require(zeallot)
 
exp <- list(
  name = 'hindcast',
  path = file.path(data_path, '$EXP_NAME$/daily_mean',
                   '$VAR_NAME$_6hourly/$VAR_NAME$_$START_DATE$.nc')
)
obs <- list(
  name = 'era5',
  path = file.path(data_path, '$EXP_NAME$/daily_mean',
                   '$VAR_NAME$_6hourly/$VAR_NAME$_$START_DATE$.nc')
)
  
c(expTAS, obsTAS) %<-% <- Load(var = temp, exp=list(exp), obs=list(obs),
           sdates = dateseq, latmin = 22, latmax = 70, 
           lonmin = -80, lonmax = 50, output = 'lonlat',
           nprocs = 1, storefreq = "daily", nmember = 1,
           method = "bilinear", grid = paste("r", grid, sep = ""))

c(expPSL, obsPSL) %<-%  <- Load(var = slp, exp=list(exp), obs=list(obs),
                sdates = dateseq, latmin = 22, latmax = 70, 
                lonmin = -80, lonmax = 50, output = 'lonlat',
                nprocs = 1, storefreq = "daily", nmember = 1,
                method = "bilinear", grid = paste("r", grid, sep = ""))

save(expTAS,obsTAS,expPSL,obsPSL, file = paste(data_path,
                        "/example_downscaling_analogs_CMCC",
                        exp$name,"_",obs$name,"_NA.Rdat",sep=""))
                        
# Or use the following links to download the file provided in 
.RData format (98MB):
https://mega.nz/file/gn50yLRL#vpGCkKUuwMbaavDkOPLwbko__pMdppf6ysz25zTwIOY
ftp://downloads.cmcc.bo.it
# then just load it in your R working environment 
load(file = "./example_downscaling_analogs_CMCChindcast_era5_NA.RData")
     
```

There should be four new elements loaded in the R working environment: `expTAS`, `obsTAS`, `expPSL`and `obsPSL`. The first two elements correspond to the experimental and observed data for temperature and the other  two are the equivalent for the SLP data. It's possible to check that they are of class `sd2v_cube` by running:


```
class(expTAS)
class(obsTAS)
class(expPSL)
class(obsPSL)
```

Loading the data using `CST_Load` allows to obtain two lists, one for the experimental data and another for the observe data, with the same elements and compatible dimensions of the data element:


```
> dim(expTAS$mod)
dataset  member   sdate   ftime     lat     lon 
      1       1      7       31     193    521
> dim(obsTAS$mod)
dataset  member   sdate   ftime     lat     lon 
      1       1      7       31     193     521
```

Latitudes and longitudes of the common grid can be saved after the interpolation:


```r
Lat <- expTAS$lat
Lon <- expTAS$lon
```
### 2.b- Load data using the example data

CSTools object `exp`, contains an element `exp$data` with dimensions:
```{r}  
#load("~/cstools/data/lonlat_data.RData")
dim(lonlat_data$exp$data)
# dataset  member   sdate   ftime     lat     lon 
#      1      15       6       3      22      53 
data <- lonlat_data
obsL <- data$obs$data
expL <- data$exp$data
lon <- data$exp$lon
lat <- data$exp$lat
dim(obsL)
dim(expL)
```

There are 15 ensemble members available in the data set, 6 starting dates and 3 forecast times, which refer to daily values in the month of November following starting dates on November 1st in the years 2010, 2011, 2012. 

```{r} 
time_obsL = as.Date(data$exp$Dates$start)

```
### 3.- Downscaling using 'CST_Load' with two variables
 First, we will prepare the arguments and data and then we will perform the three criterias of the function. For the example, select first dataset, member, stade and ftime and in 'time_expL' we will set the day (list of days) in which we want to perform the downscaling. Time_expL will be the day in which we will search for analogs in the observations, for the example set to '2001-10-28'.  The local region is set in the lon-lat of Balearic Island. Important! obsL must have 3 dimensions: time, lat, lon, with names.
 
```{r} 
dd=28
mm="10"
yy=2001
member=1
sdate=1
dataset=1
time_expL=paste0(yy,"-",mm,"-",dd)
time_obsL=as.Date(obsPSL$Dates$start)

# lat, lon vars
grid_lon<-expPSL$lon
grid_lon[grid_lon>=180]<-grid_lon[grid_lon>=180]-360
grid_lat=as.vector(sort(expPSL$lat))
region=c(min(grid_lon), max(grid_lon), min(grid_lat), max(grid_lat))

#important! obsL must have 3 dimensions: time, lat, lon, with name!
obsL_2d=obsPSL$mod[1,member,,,,] 
dim(obsL_2d) <-c(dim(obsL_2d)[2]*dim(obsL_2d)[1], length(grid_lat), length(grid_lon))
dim(obsL_2d)
names(dim(obsL_2d))[1] <- 'time'
names(dim(obsL_2d))[2] <- 'lat'
names(dim(obsL_2d))[3] <- 'lon'
dim(obsL_2d)

obsL_2dtas=obsTAS$mod[1,member,,,,]
dim(obsL_2dtas)<-c(dim(obsL_2dtas)[2]*dim(obsL_2dtas)[1], length(grid_lat), length(grid_lon))
names(dim(obsL_2dtas))[1] <- 'time'
names(dim(obsL_2dtas))[2] <- 'lat'
names(dim(obsL_2dtas))[3] <- 'lon'

expL=expPSL$mod[dataset,member,sdate,dd,,]
obsL=obsL_2d[dd,,]
expVar=expTAS$mod[dataset,member,sdate,dd,,]
obsVar=obsL_2dtas[dd,,]
dim(obsVar)
```
### 3.a - Downscaling using 'CST_Load' with two variables and Criteria 1: Large Scale pattern (distance)
```{r} 
LargeScale<- Analogs(expL=expL, 
                       obsL=obsL_2d, 
                       time_obsL=time_obsL, time_expL=time_expL,  
                       AnalogsInfo=TRUE, 
                       criteria="Large_dist",obsVar=obsL_2dtas, 
                       expVar=expVar,nAnalogs=3, lonVar=grid_lon, 
                       latVar=grid_lat)

str(LargeScale)

# plot 
layout(matrix(1:3,1,3,byrow=T))
par(mar=c(3,3,3,4))

# reference day psl
field=apply(expL, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
                field[1:(switchlon-1)]),length(grid_lat),
              length(grid_lon))
image.plot(grid_lon, grid_lat, matrix(t(field),length(grid_lon),length(grid_lat)),
           main=paste0("expSLP ",time_expL),xlab="",ylab="",
           zlim=c(min(field,na.rm=TRUE),max(field,na.rm=TRUE)))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# reference day tas
field=apply(expVar, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(grid_lon, grid_lat, matrix(t(field),length(grid_lon),length(grid_lat)),
           main=paste0("expTAS ",time_expL),xlab="",ylab="",
           zlim=c(240,315))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# analog 
ana=apply(LargeScale2a$AnalogsFields[1,,], 2, rev) 
ana=matrix(c(ana[switchlon:length(ana)],ana[1:(switchlon-1)]),length(grid_lat),length(grid_lon))

image.plot(grid_lon, grid_lat, matrix(t(ana),length(grid_lon),length(grid_lat)),
           main=paste0("Analog ",LargeScale2a$AnalogsInfo$dates[1]),xlab="",ylab="",
           zlim=c(240,315))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)
```

### 3.b - Downscaling using 'CST_Load' with two variables and Criteria 2: Local Scale pattern (distance)
```{r} 
region=c(0,5,38.5,40.5)
LocalScale<- Analogs(expL=expL, 
                       obsL=obsL_2d, 
                       time_obsL=time_obsL, time_expL=time_expL,
                       AnalogsInfo=TRUE, 
                       criteria="Local_dist",obsVar=obsL_2dtas,
                       expVar=expVar,
                       nAnalogs=10,
                       lonVar=as.vector(grid_lon),
                       latVar=as.vector(grid_lat),region=region)

str(LocalScale)

# plot 
layout(matrix(1:3,1,3,byrow=T))
par(mar=c(3,3,3,4))

# reference day psl
field=apply(expL, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(grid_lon, grid_lat, matrix(t(field),length(grid_lon),length(grid_lat)),
           main=paste0("expSLP ",time_expL),xlab="",ylab="",
           zlim=c(min(field,na.rm=TRUE),max(field,na.rm=TRUE)))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# reference day tas
field=apply(expVar, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(grid_lon, grid_lat, matrix(t(field),length(grid_lon),length(grid_lat)),
           main=paste0("expTAS ",time_expL),xlab="",ylab="",
           zlim=c(240,315))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# analog 
ana=apply(LocalScale$AnalogsFields[1,,], 2, rev) 
grid_lonlocal=grid_lon[which((grid_lon >=region[1]) & (grid_lon <= region[2]))]
grid_latlocal=grid_lat[which((grid_lat >=region[3]) & (grid_lat <= region[4]))]
image.plot(grid_lonlocal, 
           grid_latlocal, 
           matrix(t(ana),length(grid_lonlocal),length(grid_latlocal)),
           main=paste0("Analog ",LocalScale$AnalogsInfo$dates[1]),xlab="",ylab="",
           zlim=c(min(ana,na.rm=TRUE),max(ana,na.rm=TRUE)))
world(xlim=range(grid_lonlocal),ylim=range(grid_latlocal),lwd=2,interior=FALSE,add=TRUE)
```


### 3.c - Downscaling using 'CST_Load' with two variables Criteria 3: Local Scale correlation (correlation)
```{r} 
region=c(0,5,38.5,40.5)
LocalCor<- Analogs(expL=expL, 
                     obsL=obsL_2d, 
                     time_obsL=time_obsL, time_expL=time_expL, AnalogsInfo=TRUE, 
                     criteria="Local_cor",obsVar=obsL_2dtas, expVar=expVar,
                     nAnalogs=10,
                     lonVar=as.vector(grid_lon),
                     latVar=as.vector(grid_lat),region=region)

str(LocalCor)

# plot 
layout(matrix(1:3,1,3,byrow=T))
par(mar=c(3,3,3,4))

# reference day psl
field=apply(expL, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(grid_lon, grid_lat, matrix(t(field),length(grid_lon),length(grid_lat)),
           main=paste0("expSLP ",time_expL),xlab="",ylab="",
           zlim=c(min(field,na.rm=TRUE),max(field,na.rm=TRUE)))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)
# reference day tas
field=apply(expVar, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(grid_lon, grid_lat, matrix(t(field),length(grid_lon),length(grid_lat)),
           main=paste0("expTAS ",time_expL),xlab="",ylab="",
           zlim=c(240,315))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# analog 
ana=apply(LocalCor$AnalogsFields[1,,], 2, rev) 
grid_lonlocal=grid_lon[which((grid_lon >=region[1]) & (grid_lon <= region[2]))]
grid_latlocal=grid_lat[which((grid_lat >=region[3]) & (grid_lat <= region[4]))]
image.plot(grid_lonlocal, 
           grid_latlocal, 
           matrix(t(ana),length(grid_lonlocal),length(grid_latlocal)),
           main=paste0("corr Analog ",LocalCor$AnalogsInfo$dates[1]),xlab="",ylab="",
           zlim=c(min(ana,na.rm=TRUE),max(ana,na.rm=TRUE)))
world(xlim=range(grid_lonlocal),ylim=range(grid_latlocal),lwd=2,interior=FALSE,add=TRUE)
```

### 4.- Downscaling using analogs and exp$data with one single Variable
 First, we will prepare the arguments and data and then we will perform the three criterias of the function. For the example, select first dataset, member, stade and ftime and in 'time_expL' we will set the day (list of days) in which we want to perform the downscaling. Time_expL will be the day in which we will search for analogs in the observations, in this example will be '2005-10-31'. Here we will use a single variable, so we should set obsVar with the same variable as in obsL. The local region is set in the lon-lat of Balearic Island. Important! obsL must have 3 dimensions: time, lat, lon, with names.
```{r} 
#expL=expL[dataset,member,sdate,1,,] 
expL=expL[1,1,1,1,,]
time_expL="2005-10-31"
obsVar=obsL
expVar=expL

# local region of interest in our case Balearic Island
lonmin=0
lonmax=5
latmin=35
latmax=43
region=c(lonmin,lonmax,latmin,latmax)

#important! obsL must have 3 dimensions: time, lat, lon, with names
obsL_2d=obsL[1,1,,,,] 
dim(obsL_2d) <-c(dim(obsL_2d)[2]*dim(obsL_2d)[1], length(lat), length(lon))
dim(obsL_2d)
names(dim(obsL_2d))[1] <- 'time'
names(dim(obsL_2d))[2] <- 'lat'
names(dim(obsL_2d))[3] <- 'lon'

```
### 4.a - Downscaling using exp$data and Criteria 1: Large Scale pattern
```{r} 
downscale_field1 <- Analogs(expL=expL, obsL=obsL_2d, time_obsL=time_obsL,
                           time_expL = time_expL,
                           criteria="Large_dist")
str(downscale_field1)
```
### 4.b - Downscaling using exp$data and Criteria 2: Local Scale pattern (distance)
```{r}
downscale_field2 <- Analogs(expL=expL, obsL=obsL_2d, 
                           time_obsL=time_obsL,
                           time_expL = time_expL,expVar=expVar,
                           obsVar=obsVar,
                           lonVar=as.vector(lon),
                           latVar=as.vector(lat),
                           criteria="Local_dist",region=region)
str(downscale_field2)
```
### 4.c- Downscaling using exp$data and Criteria 3: Local Scale (correlation)
```{r} 
downscale_field3 <- Analogs(expL=expL, obsL=obsL_2d, 
                            expVar=expL,
                            time_obsL=time_obsL,
                            time_expL = time_expL,
                            obsVar=obsL_2d,
                            lonVar=as.vector(lon),
                            latVar=as.vector(lat),
                            criteria="Local_cor",region=region)
str(downscale_field3)
```
### 4.d- Downscaling using exp$data, nAnalogs and AnalogsInfo 
Same examples as in 3 a), b), c) but asking for information 'AnalogsInfo=TRUE' and more analogs 'nAnalogs=10'. In 3 a), b), c)  those arguments were NULL and was set by default to 'nAnalogs=1' and 'AnalogsInfo=FALSE'. 
```{r}  
# example criteria 1
downscale_field1 <- Analogs(expL=expL, obsL=obsL_2d, time_obsL=time_obsL,
                            time_expL = time_expL,
                            criteria="Large_dist",nAnalogs=10,
                            AnalogsInfo = TRUE)
# example criteria 2
downscale_field2 <- Analogs(expL=expL, obsL=obsL_2d, 
                            time_obsL=time_obsL,
                            time_expL = time_expL,expVar=expVar,
                            obsVar=obsVar,nAnalogs=10,
                            lonVar=as.vector(lon),
                            latVar=as.vector(lat),
                            criteria="Local_dist",region=region,
                            AnalogsInfo = TRUE)
# example criteria 3
downscale_field3 <- Analogs(expL=expL, obsL=obsL_2d, 
                            expVar=expL,
                            time_obsL=time_obsL,
                            time_expL = time_expL,
                            obsVar=obsL_2d,nAnalogs=10,
                            lonVar=as.vector(lon),
                            latVar=as.vector(lat),
                            criteria="Local_cor",region=region,
                            AnalogsInfo=TRUE)
# summary of the three criterias
str(downscale_field1)
str(downscale_field2)
str(downscale_field3)

```
### 4.e- Downscaling using exp$data using excludeTime
ExludeTime is set by default to Time_expL in order to find the same analog than the day of interest. If there is some interest in excluding other dates should be included in the argument 'excludeTime'
```{r}  
# example criteria 1
downscale_field1 <- Analogs(expL=expL, obsL=obsL_2d,
                            time_obsL=time_obsL,time_expL = time_expL,
                            criteria="Large_dist",nAnalogs=1,
                            excludeTime=c(time_obsL[1:3]),
                            AnalogsInfo = TRUE)
# example criteria 3
downscale_field3 <- Analogs(expL=expL, obsL=obsL_2d, 
                            expVar=expL,
                            time_obsL=time_obsL,
                            time_expL = time_expL,
                            obsVar=obsL_2d,nAnalogs=1,
                            lonVar=as.vector(lon),
                            latVar=as.vector(lat),
                            criteria="Local_cor",region=region,
                            excludeTime=c(time_obsL[1:3]),
                            AnalogsInfo=TRUE)
# summary of the three criterias
str(downscale_field1)
str(downscale_field3)

```
Note: For a better use of the function you can use the anomalies values before to apply the criterias, using CST-Anomalies of CSTools package
