---
title: "Analogs based on large scale for downscaling"
author: "M. Carmen Alvarez-Castro and M. del Mar Chaves-Montero (CMCC, Italy)"
date: "November 2020"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteEngine{knitr::knitr}
  %\VignetteIndexEntry{Analogs}
  %\usepackage[utf8]{inputenc}
---
<!--
```{r, echo = FALSE}
knitr::opts_chunk$set(eval = FALSE)
```
-->
## Downscaling seasonal forecast data using Analogs

In this example, the seasonal temperature forecasts, initialized in october, 
will be used to perform a downscaling in the Balearic Islands temperature using 
the cmcc system 3 seasonal forecasting system from the Euro-Mediterranean Center
of Climate Change (CMCC), by computing Analogs in Sea level pressure data (SLP) 
in a larger region (North Atlantic). The first step will be to load the data we 
want to downscale (i.e. cmcc) in the large region (i.e North Atlantic) for 
temperature (predictand) and SLP (predictor) and same variables and region for a
higher resolution data (ERA5). In a second step we will interpolate the model to
the resolution of ERA5. In a third step we will find the analogs using one of 
the three criterias. In a four step we will get the downscaled dataset in the 
region selected (local scale, in this case Balearic Islands)

## 1. Introduction of the function

For instance if we want to perform a temperature donwscaling in Balearic Island 
for October we will get a daily series of temperature with 1 analog per day, 
the best analog. How we define the best analog for a certain day? This function
offers three options for that:

(1) The day with the minimum Euclidean distance in a large scale field: using 
i.e. pressure or geopotencial height as variables and North Atlantic region as 
large scale region. The Atmospheric circulation pattern in the North Atlantic 
(LargeScale) has an important role in the climate in Spain (LocalScale). 
The function will find the day with the most similar pattern in atmospheric 
circulation in the database (obs, slp in ERA5) to the day of interest 
(exp,slp in model). Once the date of the best analog is found, the function 
takes the associated temperature to that day (obsVar, tas in ERA5), with a 
subset of the region of interest (Balearic Island)

(2) Same that (1) but in this case we will search for analogs in the local 
scale (Balearic Island) instead of in the large scale (North Atlantic). 
Once the date of the best analog is found, the function takes the associated 
temperature to that day (obsVar, t2m in ERA5), with a subset of the region of 
interest (Balearic Island)

(3) Same that (2) but here we will search for analogs with higher correlation 
at local scale (Balearic Island) and instead of using SLP we will use t2m. 


In particular the _Analogs Method_  uses a nonlinear approach that follows 
(**Analogs**; Yiou et al. 2013) 

An efficient implementation of Analogs is provided for CSTools by the 
`CST_Analogs()` function. 

### 1. Preliminary setup

In order to run the examples in this vignette, the *CSTools* package and some 
other support R packages need to be loaded by running:

```{r}
install.packages('CSTools')
library(CSTools)
library(s2dverification)
library(ClimProjDiags)
library(multiApply)
library(zeallot)
library(fields)
library(abind)
```
### 2. Load data 
Here we show two options for the example, a) loading data using CST_Load, and 
b) loading the experimental data included in CSTools package (lonlat_data)

### 2.a- Load data using CST_Load
The parameters defined are the initializing month and the variables:

```{r}
mth = '10'
temp = 'tas'
slp = 'psl'
```


The simulations available for this model cover the period 1993-2016. 
For ERA5 from 1979 to the present days. For this example we will just use from 
2000 to 2006, so, the starting and ending dates can be defined by running the 
following lines:


```{r}
ini <- 2000
fin <- 2006
start <- as.Date(paste(ini, mth, "01", sep = ""), "%Y%m%d")
end <- as.Date(paste(fin, mth, "01", sep = ""), "%Y%m%d")
dateseq <- format(seq(start, end, by = "year"), "%Y%m%d")
``` 


The grid in which all data will be interpolated for the downscaling should be 
also specified. The observational dataset used in this example is the ERA5. 


```{r}
grid <- '1440x721'
```

Using the `CST_Load` function from **CSTool package**, the data available in our
data store can be loaded. The following lines show how this function can be 
used. Here, the data are loaded from a previous saved `.RData` file:
Ask carmen.alvarez-castro at cmcc.it or nuria.perez at bsc.es for the data to 
run the recipe.

```{r}
data_path <-"your-data-path-here"
 
exp <- list(
  name = 'hindcast',
  path = file.path(data_path, '$EXP_NAME$/daily_mean',
                   '$VAR_NAME$_6hourly/$VAR_NAME$_$START_DATE$.nc')
)
obs <- list(
  name = 'era5',
  path = file.path(data_path, '$EXP_NAME$/daily_mean',
                   '$VAR_NAME$_6hourly/$VAR_NAME$_$START_DATE$.nc')
)
  
c(expTAS, obsTAS) <- CST_Load(var = temp, exp=list(exp), obs=list(obs),
           sdates = dateseq, latmin = 22, latmax = 70, 
           lonmin = -80, lonmax = 50, output = 'lonlat',
           nprocs = 1, storefreq = "daily", nmember = 1,
           method = "bilinear", grid = paste("r", grid, sep = ""))

c(expPSL, obsPSL) <- CST_Load(var = slp, exp=list(exp), obs=list(obs),
                sdates = dateseq, latmin = 22, latmax = 70, 
                lonmin = -80, lonmax = 50, output = 'lonlat',
                nprocs = 1, storefreq = "daily", nmember = 1,
                method = "bilinear", grid = paste("r", grid, sep = ""))

save(expTAS,obsTAS,expPSL,obsPSL, file = paste(data_path,
                        "/example_downscaling_analogs_CMCC",
                        exp$name,"_",obs$name,"_NA.Rdat",sep=""))
                        
# Or use the following links to download the file provided in .RData 
# format(98MB):
# https://mega.nz/file/gn50yLRL#vpGCkKUuwMbaavDkOPLwbko__pMdppf6ysz25zTwIOY
# or
# ftp://downloads.cmcc.bo.it
# then just load it in your R working environment 
#load(file = "./example_downscaling_analogs_CMCChindcast_era5_NA.RDat")
```

There should be four new elements loaded in the R working environment: `expTAS`,
`obsTAS`, `expPSL`and `obsPSL`. The first two elements correspond to the 
experimental and observed data for temperature and the other two are the 
equivalent for the SLP data. 

Loading the data using `CST_Load` allows to obtain two lists, one for the 
experimental data and another for the observe data, with the same elements and 
compatible dimensions of the data element:


```{r}
dim(expTAS$mod)
#dataset  member   sdate   ftime     lat     lon 
#      1       1      7       31     193    521
dim(obsTAS$mod)
#dataset  member   sdate   ftime     lat     lon 
#      1       1      7       31     193     521
```

Latitudes and longitudes of the common grid can be saved after the interpolation:

```{r}
Lat <- expTAS$lat
Lon <- expTAS$lon
```
### 2.b- Load data using the example data

CSTools object `exp`, contains an element `exp$data` with dimensions:
```{r}  
#load("~/cstools/data/lonlat_data.RData")
dim(lonlat_data$exp$data)
# dataset  member   sdate   ftime     lat     lon 
#      1      15       6       3      22      53 
data <- lonlat_data
obsL <- data$obs$data
expL <- data$exp$data
lon <- data$exp$lon
lat <- data$exp$lat
dim(obsL)
dim(expL)
```

There are 15 ensemble members available in the data set, 6 starting dates and 3 
forecast times, which refer to daily values in the month of November following 
starting dates on November 1st in the years 2010, 2011, 2012. 

```{r} 
time_obsL = as.Date(data$exp$Dates$start)
```
### 3.- Downscaling data loaded with 2.a using analogs with two variables
First, we will prepare the arguments and data and then we will perform the 
analysis with each of the three criterias of the function. For the example, 
select first dataset, member, stade and ftime and in 'time_expL' we will set the
day (list of days) in which we want to perform the downscaling. Time_expL will 
be the day in which we will search for analogs in the observations, for the 
example set to '2001-10-28'.  The local region is set in the lon-lat of Balearic
Island. Important! obsL must have 3 dimensions: time, lat, lon, with names.
 
```{r} 
dd=28
mm="10"
yy=2001
member=1
sdate=1
dataset=1
time_expL=as.Date(paste(yy, mm, dd, sep = ""), "%Y%m%d")
time_obsL=as.Date(obsPSL$Dates$start)

# lat, lon vars
grid_lon<-expPSL$lon
grid_lon[grid_lon>=180]<-grid_lon[grid_lon>=180]-360
grid_lat=sort(expPSL$lat)
region=c(min(grid_lon), max(grid_lon), min(grid_lat), max(grid_lat))

#important! obsL must have 3 dimensions: time, lat, lon, with name!
obsL_2d=obsPSL$mod[1,member,,,,] 
dim(obsL_2d) <-c(dim(obsL_2d)[2]*dim(obsL_2d)[1], length(grid_lat), 
length(grid_lon))
dim(obsL_2d)
#ftime             
#  217   193   521 
names(dim(obsL_2d))[1] <- 'time'
names(dim(obsL_2d))[2] <- 'lat'
names(dim(obsL_2d))[3] <- 'lon'
dim(obsL_2d)
#time  lat  lon 
# 217  193  521 

obsL_2dtas=obsTAS$mod[1,member,,,,]
dim(obsL_2dtas)<-c(dim(obsL_2dtas)[2]*dim(obsL_2dtas)[1], length(grid_lat), 
length(grid_lon))
names(dim(obsL_2dtas))[1] <- 'time'
names(dim(obsL_2dtas))[2] <- 'lat'
names(dim(obsL_2dtas))[3] <- 'lon'
dim(obsL_2dtas)
#time  lat  lon 
# 217  193  521 

expL=expPSL$mod[dataset,member,sdate,dd,,]
obsL=obsL_2d
expVar=expTAS$mod[dataset,member,sdate,dd,,]
obsVar=obsL_2dtas
dim(obsVar)
#time  lat  lon 
# 217  193  521 
```
### 3.a - Downscaling data loaded with 2.a using analogs with two variables and
# Criteria 1: Large Scale pattern (distance)
```{r} 
LargeScale<- Analogs(expL=expL, obsL=obsL, time_obsL=time_obsL, 
                    time_expL=time_expL,
                     AnalogsInfo=TRUE,criteria="Large_dist", nAnalogs=3,
                     obsVar=obsVar, expVar=expVar, 
                     lonVar=grid_lon, latVar=grid_lat)

str(LargeScale)

# plot 
layout(matrix(1:3,1,3,byrow=T))
par(mar=c(3,3,3,4))

# reference day psl
field=apply(expL, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
                field[1:(switchlon-1)]),length(grid_lat),
              length(grid_lon))
image.plot(sort(grid_lon), grid_lat, matrix(t(field),length(grid_lon),
length(grid_lat)),
           main=paste0("expSLP ",time_expL),xlab="",ylab="",
           zlim=c(min(field,na.rm=TRUE),max(field,na.rm=TRUE)))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# reference day tas
field=apply(expVar, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(sort(grid_lon), grid_lat, matrix(t(field),length(grid_lon),
length(grid_lat)),
           main=paste0("expTAS ",time_expL),xlab="",ylab="",
           zlim=c(240,315))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# analog 
ana=apply(LargeScale$AnalogsFields[1,,], 2, rev) 
ana=matrix(c(ana[switchlon:length(ana)],ana[1:(switchlon-1)]),length(grid_lat),
length(grid_lon))

image.plot(sort(grid_lon), grid_lat, matrix(t(ana),length(grid_lon),
length(grid_lat)),
           main=paste0("Analog ",LargeScale$AnalogsInfo$dates[1]),
           xlab="",ylab="",
           zlim=c(240,315))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)
```

### 3.b - Downscaling data loaded with 2.a using analogs with two variables and 
# Criteria 2: Local Scale pattern (distance)
```{r} 
region=c(0,5,38.5,40.5)
LocalScale<- Analogs(expL=expL, obsL=obsL, time_obsL=time_obsL, 
time_expL=time_expL,
                     AnalogsInfo=TRUE, 
                     criteria="Local_dist", nAnalogs=50, 
                     obsVar=obsVar, expVar=expVar,
                     lonVar=as.vector(grid_lon), latVar=as.vector(grid_lat),
                     region=region)

str(LocalScale)

# plot 
layout(matrix(1:3,1,3,byrow=T))
par(mar=c(3,3,3,4))

# reference day psl
field=apply(expL, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(sort(grid_lon), grid_lat, matrix(t(field),length(grid_lon),
length(grid_lat)),
           main=paste0("expSLP ",time_expL),xlab="",ylab="",
           zlim=c(min(field,na.rm=TRUE),max(field,na.rm=TRUE)))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# reference day tas
field=apply(expVar, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(sort(grid_lon), grid_lat, matrix(t(field),length(grid_lon),
length(grid_lat)),
           main=paste0("expTAS ",time_expL),xlab="",ylab="",
           zlim=c(240,315))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# analog 
ana=apply(LocalScale$AnalogsFields[1,,], 2, rev) 
grid_lonlocal=grid_lon[which((grid_lon >=region[1]) & (grid_lon <= region[2]))]
grid_latlocal=grid_lat[which((grid_lat >=region[3]) & (grid_lat <= region[4]))]
image.plot(grid_lonlocal, 
           grid_latlocal, 
           matrix(t(ana),length(grid_lonlocal),length(grid_latlocal)),
           main=paste0("Analog ",LocalScale$AnalogsInfo$dates[1]),xlab="",
           ylab="",
           zlim=c(min(ana,na.rm=TRUE),max(ana,na.rm=TRUE)))
world(xlim=range(grid_lonlocal),ylim=range(grid_latlocal),lwd=2,interior=FALSE,
add=TRUE)
```


### 3.c - Downscaling data loaded with 2.a using analogs with two variables 
# Criteria 3: Local Scale correlation (correlation)
```{r} 
region=c(0,5,38.5,40.5)
LocalCor<- Analogs(expL=expL, obsL=obsL, time_obsL=time_obsL, 
time_expL=time_expL,
                   AnalogsInfo=TRUE, 
                   criteria="Local_cor", nAnalogs=50,
                   obsVar=obsL_2dtas, expVar=expVar,
                   lonVar=as.vector(grid_lon), latVar=as.vector(grid_lat),
                   region=region)

str(LocalCor)

# plot 
layout(matrix(1:3,1,3,byrow=T))
par(mar=c(3,3,3,4))

# reference day psl
field=apply(expL, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(sort(grid_lon), grid_lat, matrix(t(field),length(grid_lon),
length(grid_lat)),
           main=paste0("expSLP ",time_expL),xlab="",ylab="",
           zlim=c(min(field,na.rm=TRUE),max(field,na.rm=TRUE)))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)
# reference day tas
field=apply(expVar, 2, rev)
switchlon=length(field)-(length(which(grid_lon<0))*length(grid_lat))
field=matrix(c(field[switchlon:length(field)],
               field[1:(switchlon-1)]),length(grid_lat),
             length(grid_lon))
image.plot(sort(grid_lon), grid_lat, matrix(t(field),length(grid_lon),
length(grid_lat)),
           main=paste0("expTAS ",time_expL),xlab="",ylab="",
           zlim=c(240,315))
world(xlim=range(grid_lon),ylim=range(grid_lat),lwd=2,interior=FALSE,add=TRUE)

# analog 
ana=apply(LocalCor$AnalogsFields[1,,], 2, rev) 
grid_lonlocal=grid_lon[which((grid_lon >=region[1]) & (grid_lon <= region[2]))]
grid_latlocal=grid_lat[which((grid_lat >=region[3]) & (grid_lat <= region[4]))]
image.plot(grid_lonlocal, 
           grid_latlocal, 
           matrix(t(ana),length(grid_lonlocal),length(grid_latlocal)),
           main=paste0("corr Analog ",LocalCor$AnalogsInfo$dates[1]),
           xlab="",ylab="",
           zlim=c(min(ana,na.rm=TRUE),max(ana,na.rm=TRUE)))
world(xlim=range(grid_lonlocal),ylim=range(grid_latlocal),lwd=2,interior=FALSE,
add=TRUE)
```

### 4.- Downscaling data loaded with 2.b using analogs and exp$data with one 
# single Variable
 First, we will prepare the arguments and data and then we will perform the 
 three criterias of the function. For the example, select first dataset, member,
 stade and ftime and in 'time_expL' we will set the day (list of days) in which 
 we want to perform the downscaling. Time_expL will be the day in which we will 
 search for analogs in the observations, in this example will be '2005-10-31'. 
 Here we will use a single variable, so we should set obsVar with the same 
 variable as in obsL. The local region is set in the lon-lat of Balearic Island.
 Important! obsL must have 3 dimensions: time, lat, lon, with names.
```{r} 
# Remember to load the data following point 2.b!
#expL=expL[dataset,member,sdate,1,,] 
expL=expL[1,1,1,1,,]
time_expL="2005-10-31"
expVar=expL
time_obsL=lonlat_data$exp$Dates$start

# local region of interest in our case Balearic Island
lonmin=0
lonmax=5
latmin=35
latmax=43
region=c(lonmin,lonmax,latmin,latmax)

#important! obsL must have 3 dimensions: time, lat, lon, with names
obsL_2d=obsL[1,1,,,,] 
dim(obsL_2d) <-c(dim(obsL_2d)[2]*dim(obsL_2d)[1], length(lat), length(lon))
dim(obsL_2d)
names(dim(obsL_2d))[1] <- 'time'
names(dim(obsL_2d))[2] <- 'lat'
names(dim(obsL_2d))[3] <- 'lon'
obsL<-obsL_2d
obsVar=obsL

```
### 4.a - Downscaling data loaded with 2-b using analogs with 
# criteria 1: Large Scale pattern
```{r} 
downscale_field1 <- Analogs(expL=expL, obsL=obsL, 
                            time_obsL=time_obsL, 
                            time_expL = time_expL,
                            criteria="Large_dist")
str(downscale_field1)
```
### 4.b - Downscaling data loaded with 2-b using analogs with 
# criteria 2: Local Scale pattern (distance)
```{r}
downscale_field2 <- Analogs(expL=expL, obsL=obsL, 
                           time_obsL=time_obsL,
                           time_expL = time_expL,
                           criteria="Local_dist",
                           expVar=expVar, obsVar=obsVar,
                           lonVar=as.vector(lon), latVar=as.vector(lat),
                           region=region)
str(downscale_field2)
```
### 4.c- Downscaling using exp$data and Criteria 3: Local Scale (correlation)
```{r} 
downscale_field3 <- Analogs(expL=expL, obsL=obsL, 
                            expVar=expL,
                            time_obsL=time_obsL,
                            time_expL = time_expL,
                            obsVar=obsVar,
                            lonVar=as.vector(lon),
                            latVar=as.vector(lat),
                            criteria="Local_cor",region=region)
str(downscale_field3)
```
### 4.d- Downscaling using exp$data, nAnalogs and AnalogsInfo 
Same examples as in 3 a), b), c) but asking for information 'AnalogsInfo=TRUE'
and more analogs 'nAnalogs=10'. In 3 a), b), c)  those arguments were NULL and 
was set by default to 'nAnalogs=1' and 'AnalogsInfo=FALSE'. 
```{r}  
# example criteria 1
downscale_field1 <- Analogs(expL=expL, obsL=obsL_2d, time_obsL=time_obsL,
                            time_expL = time_expL,
                            criteria="Large_dist",nAnalogs=10,
                            AnalogsInfo = TRUE)
# example criteria 2
downscale_field2 <- Analogs(expL=expL, obsL=obsL_2d, 
                            time_obsL=time_obsL,
                            time_expL = time_expL,
                            AnalogsInfo = TRUE,
                            criteria="Local_dist", nAnalogs=10,
                            expVar=expVar, obsVar=obsVar,
                            lonVar=as.vector(lon),latVar=as.vector(lat),
                            region=region)
# example criteria 3
downscale_field3 <- Analogs(expL=expL, obsL=obsL_2d, 
                            expVar=expL,
                            time_obsL=time_obsL,
                            time_expL = time_expL,
                            AnalogsInfo=TRUE,
                            criteria="Local_cor",nAnalogs=10,
                            obsVar=obsL_2d,
                            lonVar=as.vector(lon),latVar=as.vector(lat),
                            region=region)                            
# summary of the three criterias
str(downscale_field1)
str(downscale_field2)
str(downscale_field3)

```
### 4.e- Downscaling using exp$data using excludeTime
ExludeTime is set by default to Time_expL in order to find the same analog than
the day of interest. If there is some interest in excluding other dates should 
be included in the argument 'excludeTime'
```{r}  
# example criteria 1
downscale_field1 <- Analogs(expL=expL, obsL=obsL_2d,
                            time_obsL=time_obsL,time_expL = time_expL,
                            criteria="Large_dist",nAnalogs=10,
                            excludeTime=time_obsL[5],
                            AnalogsInfo = TRUE)
# example criteria 3
downscale_field3 <- Analogs(expL=expL, obsL=obsL_2d, 
                            expVar=expL,
                            time_obsL=time_obsL,
                            time_expL = time_expL,
                            obsVar=obsL_2d,
                            AnalogsInfo=TRUE, 
                            criteria="Local_cor", nAnalogs=10,
                            lonVar=as.vector(lon),
                            latVar=as.vector(lat),region=region,
                            excludeTime=c(time_obsL[1:3]) )
# summary of the three criterias
str(downscale_field1)
str(downscale_field3)

```
Note: For a better use of the function you can use the anomalies values before 
to apply the criterias, using CST-Anomalies of CSTools package
