---
title: "Analogs based on large scale for downscaling"
author: "M. Carmen Alvarez-Castro and M. del Mar Chaves-Montero (CMCC, Italy)"
date: "November 2020"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteEngine{knitr::knitr}
  %\VignetteIndexEntry{Analogs}
  %\usepackage[utf8]{inputenc}
---
<!--
```{r, echo = FALSE}
knitr::opts_chunk$set(eval = FALSE)
```
-->
## Downscaling seasonal forecast data using Analogs

In this example, the seasonal temperature forecasts, initialized in october, will be used to perform a downscaling in the Balearic Islands temperature using the cmcc system 3 seasonal forecasting system from the Euro-Mediterranean Center of Climate Change (CMCC), by computing Analogs in Sea level pressure data (SLP) in a larger region (North Atlantic). The first step will be to load the data we want to downscale (i.e. cmcc) in the large region (i.e North Atlantic) for temperature (predictand) and SLP (predictor) and same variables and region for a higher resolution data (ERA5). In a second step we will interpolate the model to the resolution of ERA5. In a third step we will find the analogs using one of the three criterias. In a four step we will get the downscaled dataset in the region selected (local scale, in this case Balearic Islands)

## 1. Introduction of the function

For instance if we want to perform a temperature donwscaling in Balearic Island for October we will get a daily series of temperature with 1 analog per day, the best analog. How we define the best analog for a certain day? This function offers three options for that:

(1) The day with the minimum Euclidean distance in a large scale field: using i.e. pressure or geopotencial height as variables and North Atlantic region as large scale region. The Atmospheric circulation pattern in the North Atlantic (LargeScale) has an important role in the climate in Spain (LocalScale). The function will find the day with the most similar pattern in atmospheric circulation in the database (obs, slp in ERA5) to the day of interest (exp,slp in model). Once the date of the best analog is found, the function takes the associated temperature to that day (obsVar, tas in ERA5), with a subset of the region of interest (Balearic Island)

(2) Same that (1) but in this case we will search for analogs in the local scale (Balearic Island) instead of in the large scale (North Atlantic). Once the date of the best analog is found, the function takes the associated temperature to that day (obsVar, t2m in ERA5), with a subset of the region of interest (Balearic Island)

(3) Same that (2) but here we will search for analogs with higher correlation at local scale (Balearic Island) and instead of using SLP we will use t2m. 


In particular the _Analogs Method_  uses a nonlinear approach that follows (**Analogs**; Yiou et al. 2013) 

An efficient implementation of Analogs is provided for CSTools by the `CST_Analogs()` function. 

### 1. Preliminary setup

In order to run the examples in this vignette, the *CSTools* package and some other support R packages need to be loaded by running:

```{r}
install.packages('CSTools')
library(CSTools)
library(s2dverification)
library(ClimProjDiags)
library(multiApply)
library(zeallot)
library(fields)
library(abind)
```
### 2. Load data 
Here we show two options for the example, a) loading data using CST_Load, and b) loading the experimental data included in CSTools package (lonlat_data)

### 2.a- Load data using CST_Load
The parameters defined are the initializing month and the variables:

```{r}
mth = '10'
temp = 'tas'
slp = 'psl'
```


The simulations available for this model cover the period 1993-2016. For ERA5 from 1979 to the present days. For this example we will just use from 2000 to 2006, so, the starting and ending dates can be defined by running the following lines:


```{r}
ini <- 2000
fin <- 2006
start <- as.Date(paste(ini, mth, "01", sep = ""), "%Y%m%d")
end <- as.Date(paste(fin, mth, "01", sep = ""), "%Y%m%d")
dateseq <- format(seq(start, end, by = "year"), "%Y%m%d")
``` 


The grid in which all data will be interpolated for the downscaling should be also specified. The observational dataset used in this example is the ERA5. 


```{r}
grid <- '1440x721'
```

Using the `CST_Load` function from **CSTool package**, the data available in our data store can be loaded. The following lines show how this function can be used. Here, the data are loaded from a previous saved `.RData` file:
Ask carmen.alvarez-castro at cmcc.it or nuria.perez at bsc.es for the data to run the recipe.

```{r}
data_path <-"your-data-path-here"
 
exp <- list(
  name = 'hindcast',
  path = file.path(data_path, '$EXP_NAME$/daily_mean',
                   '$VAR_NAME$_6hourly/$VAR_NAME$_$START_DATE$.nc')
)
obs <- list(
  name = 'era5',
  path = file.path(data_path, '$EXP_NAME$/daily_mean',
                   '$VAR_NAME$_6hourly/$VAR_NAME$_$START_DATE$.nc')
)
  
c(expTAS, obsTAS) <- CST_Load(var = temp, exp=list(exp), obs=list(obs),
           sdates = dateseq, latmin = 22, latmax = 70, 
           lonmin = -80, lonmax = 50, output = 'lonlat',
           nprocs = 1, storefreq = "daily", nmember = 1,
           method = "bilinear", grid = paste("r", grid, sep = ""))

c(expPSL, obsPSL) <- CST_Load(var = slp, exp=list(exp), obs=list(obs),
                sdates = dateseq, latmin = 22, latmax = 70, 
                lonmin = -80, lonmax = 50, output = 'lonlat',
                nprocs = 1, storefreq = "daily", nmember = 1,
                method = "bilinear", grid = paste("r", grid, sep = ""))

save(expTAS,obsTAS,expPSL,obsPSL, file = paste(data_path,
                        "/example_downscaling_analogs_CMCC",
                        exp$name,"_",obs$name,"_NA.Rdat",sep=""))
                        
# Or use the following links to download the file provided in .RData format (98MB):
# https://mega.nz/file/gn50yLRL#vpGCkKUuwMbaavDkOPLwbko__pMdppf6ysz25zTwIOY
# or
# ftp://downloads.cmcc.bo.it
# then just load it in your R working environment 
#load(file = "./example_downscaling_analogs_CMCChindcast_era5_NA.RDat")
```

There should be four new elements loaded in the R working environment: `expTAS`, `obsTAS`, `expPSL`and `obsPSL`. The first two elements correspond to the experimental and observed data for temperature and the other two are the equivalent for the SLP data. 

Loading the data using `CST_Load` allows to obtain two lists, one for the experimental data and another for the observe data, with the same elements and compatible dimensions of the data element:


```{r}
dim(expTAS$mod)
#dataset  member   sdate   ftime     lat     lon 
#      1       1      7       31     193    521
dim(obsTAS$mod)
#dataset  member   sdate   ftime     lat     lon 
#      1       1      7       31     193     521
```

Latitudes and longitudes of the common grid can be saved after the interpolation:

```{r}
Lat <- expTAS$lat
Lon <- expTAS$lon
```
### 2.b- Load data using the example data

CSTools object `exp`, contains an element `exp$data` with dimensions:
```{r}  
#load("~/cstools/data/lonlat_data.RData")
dim(lonlat_data$exp$data)
# dataset  member   sdate   ftime     lat     lon 
#      1      15       6       3      22      53 
data <- lonlat_data
obsL <- data$obs$data
expL <- data$exp$data
lon <- data$exp$lon
lat <- data$exp$lat
dim(obsL)
dim(expL)
```

There are 15 ensemble members available in the data set, 6 starting dates and 3 forecast times, which refer to daily values in the month of November following starting dates on November 1st in the years 2010, 2011, 2012. 

```{r} 
time_obsL = as.Date(data$exp$Dates$start)
```
### 3.- Downscaling data loaded with 2.a using analogs with two variables
First, we will prepare the arguments and data and then we will perform the analysis with each of the three criterias of the function. For the example, select first dataset, member, stade and ftime and in 'time_expL' we will set the day (list of days) in which we want to perform the downscaling. Time_expL will be the day in which we will search for analogs in the observations, for the example set to '2001-10-28'.  The local region is set in the lon-lat of Balearic Island. Important! obsL must have 3 dimensions: time, lat, lon, with names.
 
```{r} 
dd=28
mm="10"
yy=2001
member=1
sdate=1
dataset=1
time_expL=as.Date(paste(yy, mm, dd, sep = ""), "%Y%m%d")
time_obsL=as.Date(obsPSL$Dates$start)

# lat, lon vars
grid_lon<-expPSL$lon
grid_lon[grid_lon>=180]<-grid_lon[grid_lon>=180]-360
grid_lat=sort(expPSL$lat)
region=c(min(grid_lon), max(grid_lon), min(grid_lat), max(grid_lat))

#important! obsL must have 3 dimensions: time, lat, lon, with name!
obsL_2d=obsPSL$mod[1,member,,,,] 
dim(obsL_2d) <-c(dim(obsL_2d)[2]*dim(obsL_2d)[1], length(grid_lat), length(grid_lon))
dim(obsL_2d)
#ftime             
#  217   193   521 
names(dim(obsL_2d))[1] <- 'time'
names(dim(obsL_2d))[2] <- 'lat'
names(dim(obsL_2d))[3] <- 'lon'
dim(obsL_2d)
#time  lat  lon 
# 217  193  521 

obsL_2dtas=obsTAS$mod[1,member,,,,]
dim(obsL_2dtas)<-c(dim(obsL_2dtas)[2]*dim(obsL_2dtas)[1], length(grid_lat), length(grid_lon))
names(dim(obsL_2dtas))[1] <- 'time'
names(dim(obsL_2dtas))[2] <- 'lat'
names(dim(obsL_2dtas))[3] <- 'lon'
dim(obsL_2dtas)
#time  lat  lon 
# 217  193  521 

expL=expPSL$mod[dataset,member,sdate,dd,,]
obsL=obsL_2d
expVar=expTAS$mod[dataset,member,sdate,dd,,]
obsVar=obsL_2dtas
dim(obsVar)
#time  lat  lon 
# 217  193  521 
```
### 3.a - Downscaling data loaded with 2.a using analogs with two variables and Criteria 1: Large Scale pattern (distance)
```{r} 
LargeScale<- Analogs(expL=expL, obsL=obsL, time_obsL=time_obsL, time_expL=time_expL,
                     AnalogsInfo=TRUE,criteria="Large_dist", nAnalogs=3,
                     obsVar=obsVar, expVar=expVar, 
                     lonVar=grid_lon, latVar=grid_lat)

str(LargeScale)

# plot 
layout(matrix(1:3,1,3,byrow=T))
par(mar=c(3,3,3,4))

# reference day psl
field=apply(expL, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
                field[1:(switchlon-1)]),length(grid_lat),
              length(grid_lon))
image.plot(sort(grid_lon), grid_lat, matrix(t(field),length(grid_lon),length(grid_lat)),
           main=paste0("expSLP ",time_expL),xlab="",ylab="",
           zlim=c(min(field,na.rm=TRUE),max(field,na.rm=TRUE)))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# reference day tas
field=apply(expVar, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(sort(grid_lon), grid_lat, matrix(t(field),length(grid_lon),length(grid_lat)),
           main=paste0("expTAS ",time_expL),xlab="",ylab="",
           zlim=c(240,315))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# analog 
ana=apply(LargeScale$AnalogsFields[1,,], 2, rev) 
ana=matrix(c(ana[switchlon:length(ana)],ana[1:(switchlon-1)]),length(grid_lat),length(grid_lon))

image.plot(sort(grid_lon), grid_lat, matrix(t(ana),length(grid_lon),length(grid_lat)),
           main=paste0("Analog ",LargeScale$AnalogsInfo$dates[1]),xlab="",ylab="",
           zlim=c(240,315))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)
```

### 3.b - Downscaling data loaded with 2.a using analogs with two variables and Criteria 2: Local Scale pattern (distance)
```{r} 
region=c(0,5,38.5,40.5)
LocalScale<- Analogs(expL=expL, obsL=obsL, time_obsL=time_obsL, time_expL=time_expL,
                     AnalogsInfo=TRUE, 
                     criteria="Local_dist", nAnalogs=50, 
                     obsVar=obsVar, expVar=expVar,
                     lonVar=as.vector(grid_lon), latVar=as.vector(grid_lat),region=region)

str(LocalScale)

# plot 
layout(matrix(1:3,1,3,byrow=T))
par(mar=c(3,3,3,4))

# reference day psl
field=apply(expL, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(sort(grid_lon), grid_lat, matrix(t(field),length(grid_lon),length(grid_lat)),
           main=paste0("expSLP ",time_expL),xlab="",ylab="",
           zlim=c(min(field,na.rm=TRUE),max(field,na.rm=TRUE)))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# reference day tas
field=apply(expVar, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(sort(grid_lon), grid_lat, matrix(t(field),length(grid_lon),length(grid_lat)),
           main=paste0("expTAS ",time_expL),xlab="",ylab="",
           zlim=c(240,315))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# analog 
ana=apply(LocalScale$AnalogsFields[1,,], 2, rev) 
grid_lonlocal=grid_lon[which((grid_lon >=region[1]) & (grid_lon <= region[2]))]
grid_latlocal=grid_lat[which((grid_lat >=region[3]) & (grid_lat <= region[4]))]
image.plot(grid_lonlocal, 
           grid_latlocal, 
           matrix(t(ana),length(grid_lonlocal),length(grid_latlocal)),
           main=paste0("Analog ",LocalScale$AnalogsInfo$dates[1]),xlab="",ylab="",
           zlim=c(min(ana,na.rm=TRUE),max(ana,na.rm=TRUE)))
world(xlim=range(grid_lonlocal),ylim=range(grid_latlocal),lwd=2,interior=FALSE,add=TRUE)
```


### 3.c - Downscaling data loaded with 2.a using analogs with two variables Criteria 3: Local Scale correlation (correlation)
```{r} 
region=c(0,5,38.5,40.5)
LocalCor<- Analogs(expL=expL, obsL=obsL, time_obsL=time_obsL, time_expL=time_expL,
                   AnalogsInfo=TRUE, 
                   criteria="Local_cor", nAnalogs=50,
                   obsVar=obsL_2dtas, expVar=expVar,
                   lonVar=as.vector(grid_lon), latVar=as.vector(grid_lat),region=region)

str(LocalCor)

# plot 
layout(matrix(1:3,1,3,byrow=T))
par(mar=c(3,3,3,4))

# reference day psl
field=apply(expL, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(sort(grid_lon), grid_lat, matrix(t(field),length(grid_lon),length(grid_lat)),
           main=paste0("expSLP ",time_expL),xlab="",ylab="",
           zlim=c(min(field,na.rm=TRUE),max(field,na.rm=TRUE)))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)
# reference day tas
field=apply(expVar, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(sort(grid_lon), grid_lat, matrix(t(field),length(grid_lon),length(grid_lat)),
           main=paste0("expTAS ",time_expL),xlab="",ylab="",
           zlim=c(240,315))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# analog 
ana=apply(LocalCor$AnalogsFields[1,,], 2, rev) 
grid_lonlocal=grid_lon[which((grid_lon >=region[1]) & (grid_lon <= region[2]))]
grid_latlocal=grid_lat[which((grid_lat >=region[3]) & (grid_lat <= region[4]))]
image.plot(grid_lonlocal, 
           grid_latlocal, 
           matrix(t(ana),length(grid_lonlocal),length(grid_latlocal)),
           main=paste0("corr Analog ",LocalCor$AnalogsInfo$dates[1]),xlab="",ylab="",
           zlim=c(min(ana,na.rm=TRUE),max(ana,na.rm=TRUE)))
world(xlim=range(grid_lonlocal),ylim=range(grid_latlocal),lwd=2,interior=FALSE,add=TRUE)
```

### 4.- Downscaling data loaded with 2.b using analogs and exp$data with one single Variable
 First, we will prepare the arguments and data and then we will perform the three criterias of the function. For the example, select first dataset, member, stade and ftime and in 'time_expL' we will set the day (list of days) in which we want to perform the downscaling. Time_expL will be the day in which we will search for analogs in the observations, in this example will be '2005-10-31'. Here we will use a single variable, so we should set obsVar with the same variable as in obsL. The local region is set in the lon-lat of Balearic Island. Important! obsL must have 3 dimensions: time, lat, lon, with names.
```{r} 
# Remember to load the data following point 2.b!
#expL=expL[dataset,member,sdate,1,,] 
expL=expL[1,1,1,1,,]
time_expL="2005-10-31"
expVar=expL
time_obsL=lonlat_data$exp$Dates$start

# local region of interest in our case Balearic Island
lonmin=0
lonmax=5
latmin=35
latmax=43
region=c(lonmin,lonmax,latmin,latmax)

#important! obsL must have 3 dimensions: time, lat, lon, with names
obsL_2d=obsL[1,1,,,,] 
dim(obsL_2d) <-c(dim(obsL_2d)[2]*dim(obsL_2d)[1], length(lat), length(lon))
dim(obsL_2d)
names(dim(obsL_2d))[1] <- 'time'
names(dim(obsL_2d))[2] <- 'lat'
names(dim(obsL_2d))[3] <- 'lon'
obsL<-obsL_2d
obsVar=obsL

```
### 4.a - Downscaling data loaded with 2-b using analogs with criteria 1: Large Scale pattern
```{r} 
downscale_field1 <- Analogs(expL=expL, obsL=obsL, 
                            time_obsL=time_obsL, 
                            time_expL = time_expL,
                            criteria="Large_dist")
str(downscale_field1)
```
### 4.b - Downscaling data loaded with 2-b using analogs with criteria 2: Local Scale pattern (distance)
```{r}
downscale_field2 <- Analogs(expL=expL, obsL=obsL, 
                           time_obsL=time_obsL,
                           time_expL = time_expL,
                           criteria="Local_dist",
                           expVar=expVar, obsVar=obsVar,
                           lonVar=as.vector(lon), latVar=as.vector(lat),region=region)
str(downscale_field2)
```
### 4.c- Downscaling using exp$data and Criteria 3: Local Scale (correlation)
```{r} 
downscale_field3 <- Analogs(expL=expL, obsL=obsL, 
                            expVar=expL,
                            time_obsL=time_obsL,
                            time_expL = time_expL,
                            obsVar=obsVar,
                            lonVar=as.vector(lon),
                            latVar=as.vector(lat),
                            criteria="Local_cor",region=region)
str(downscale_field3)
```
### 4.d- Downscaling using exp$data, nAnalogs and AnalogsInfo 
Same examples as in 3 a), b), c) but asking for information 'AnalogsInfo=TRUE' and more analogs 'nAnalogs=10'. In 3 a), b), c)  those arguments were NULL and was set by default to 'nAnalogs=1' and 'AnalogsInfo=FALSE'. 
```{r}  
# example criteria 1
downscale_field1 <- Analogs(expL=expL, obsL=obsL_2d, time_obsL=time_obsL,
                            time_expL = time_expL,
                            criteria="Large_dist",nAnalogs=10,
                            AnalogsInfo = TRUE)
# example criteria 2
downscale_field2 <- Analogs(expL=expL, obsL=obsL_2d, 
                            time_obsL=time_obsL,
                            time_expL = time_expL,
                            AnalogsInfo = TRUE,
                            criteria="Local_dist", nAnalogs=10,
                            expVar=expVar, obsVar=obsVar,
                            lonVar=as.vector(lon),latVar=as.vector(lat),region=region)
# example criteria 3
downscale_field3 <- Analogs(expL=expL, obsL=obsL_2d, 
                            expVar=expL,
                            time_obsL=time_obsL,
                            time_expL = time_expL,
                            AnalogsInfo=TRUE,
                            criteria="Local_cor",nAnalogs=10,
                            obsVar=obsL_2d,
                            lonVar=as.vector(lon),latVar=as.vector(lat),region=region)                            
# summary of the three criterias
str(downscale_field1)
str(downscale_field2)
str(downscale_field3)

```
### 4.e- Downscaling using exp$data using excludeTime
ExludeTime is set by default to Time_expL in order to find the same analog than the day of interest. If there is some interest in excluding other dates should be included in the argument 'excludeTime'
```{r}  
# example criteria 1
downscale_field1 <- Analogs(expL=expL, obsL=obsL_2d,
                            time_obsL=time_obsL,time_expL = time_expL,
                            criteria="Large_dist",nAnalogs=10,
                            excludeTime=time_obsL[5],
                            AnalogsInfo = TRUE)
# example criteria 3
downscale_field3 <- Analogs(expL=expL, obsL=obsL_2d, 
                            expVar=expL,
                            time_obsL=time_obsL,
                            time_expL = time_expL,
                            obsVar=obsL_2d,
                            AnalogsInfo=TRUE, 
                            criteria="Local_cor", nAnalogs=10,
                            lonVar=as.vector(lon),
                            latVar=as.vector(lat),region=region,
                            excludeTime=c(time_obsL[1:3]) )
# summary of the three criterias
str(downscale_field1)
str(downscale_field3)

```
Note: For a better use of the function you can use the anomalies values before to apply the criterias, using CST-Anomalies of CSTools package
