Heatwave and coldwave duration
==============================

Heatwave and coldwave duration is the total duration of "extreme spells"; the total number of days in a season where the temperature exceeds a threshold over a minimum number of consecutive days. Other tools for computing such events are also available, but the novelty of this tool is that the users can select their own thresholds (based on quantiles) and minimum duration of an event. The duration of heatwaves and coldwaves helps to understand potential changes in energy demand.

*Note: This vignette has been written to process and plot the output of an ESMValTool namelist configured with a specific set of parameters. If you wish to run and plot the outputs for another set of parameters, you need to re-run the corresponding namelist with the desired parameters, interpolate the output NetCDF files to a common grid, and modify the vignette code accordingly.*

The following example will compute the heatwaves and coldwaves duration in the North Atlantic - European Sector [20ºW-50ºE; 30ºN-80ºN] using the CMIP5 projection for the 8.5 scenario.

### 1- Load dependencies
This example requires the following system libraries:


- libssl-dev
- libnecdf-dev
- cdo


You will need to install specific versions of certain R packages as follows:
```r
library(devtools)
install_git('https://earth.bsc.es/gitlab/es/startR', branch = 'develop-hotfixes-0.0.2')
```

Five functions should be loaded by running the following lines in R, until integrated into the new magic.bsc package.

```r
source('https://earth.bsc.es/gitlab/nperez/magic.bsc/raw/develop-bugfixes-0.0.0/R/Threshold.R')
source('https://earth.bsc.es/gitlab/nperez/magic.bsc/raw/develop-bugfixes-0.0.0/R/SeasonSelect.R')
source('https://earth.bsc.es/gitlab/es/s2dverification/raw/develop-Magic/vignettes/heatcoldwaves/WaveDuration.R')
source('https://earth.bsc.es/gitlab/es/s2dverification/raw/develop-Magic/vignettes/heatcoldwaves/WaveIntensity.R')
source('https://earth.bsc.es/gitlab/es/s2dverification/raw/develop-Magic/vignettes/heatcoldwaves/WaveStart.R')
```


All the other R packages involved can be installed directly from CRAN and loaded as follows:
```r
library(s2dverification)
library(startR)
library(multiApply)
library(ggplot2)
library(climdex.pcic)
library(parallel)
```

### 2- Heatwaves

The heatwaves duration is the number of consecutive days for which the maximum temperature is exceeding a threshold over a minimum number of days during summer.

The next image shows an example of 17 days heatwave duration in a year corresponding to the red bars. Five days is the minimum number of consecutive days considered in the example.

<img src = 'heatWAVE_duration_def.png' />

#### 2.1- Defining heatwaves threshold

In this example, the threshold is defined as the 90th percentile of maximum temperature during the period 1971-2000. 

The historical simulations for the CMIP5 model should be loaded by the function `Start` from `s2dverification` package:


```r
var0 <- 'tasmax'
start_climatology <- '1971-01-01'
end_climatology <- '2000-12-31'
lat.min <- 30;  lat.max <- 80
lon.min <- -20; lon.max <- 50
climatology_filenames <- paste0("/esarchive/scratch/pbretonn/cmip5-cp4cds/historical/day/tasmax/",
                                "tasmax_day_IPSL-CM5A-LR_historical_r1i1p1_19500101-20051231.nc")

reference_data <- Start(model = climatology_filenames, var = var0, var_var = 'var_names', 
                        time = values(list(as.POSIXct(start_climatology), as.POSIXct(end_climatology))),
                        lat = values(list(lat.min, lat.max)), lon = values(list(lon.min, lon.max)), 
                        lon_var = 'lon', lon_reorder = CircularSort(0, 360), return_vars = list(time = 'model', 
                        lon = 'model', lat = 'model'), retrieve = TRUE)
metadata <- attributes(reference_data)
```


The summer data can be picked by `SeasonSelect` function from the `magic.bsc` package.

```r
reference_data <- SeasonSelect(reference_data, seasons = 'JJA')
```

To define the heatwave it is necessary to select the maximum temperature threshold that must be exceeded. In this example, the 90 th percentile of the maximum temperature during the reference period is selected. This calculation is performed using `Threshold` function from `magic.bsc` package.

```r
quantile <- 0.9
base_range <- as.numeric(c(substr(start_climatology, 1, 4), substr(end_climatology, 1, 4)))
thresholds <- Threshold(reference_data, base_range = as.numeric(base_range), 
                        qtiles = quantile, ncores = detectCores() -1)[152:243,,]
names(dim(thresholds)) <- c('jdays', 'lat', 'lon')                    
```

#### 2.2- Heatwaves projection

Considering the 8.5 rcp scenario of the IPSL-CM5A-LR model during the period 2020-2040, the data is loaded using the function `Start`.


```r
start_model <- '2020-01-01'
end_model <- '2040-12-31'
model_filenames <- paste0("/esarchive/scratch/pbretonn/cmip5-cp4cds/",
                          "rcp85/day/tasmax/tasmax_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.nc")
rcp_data <- Start(model = model_filenames, var = var0, var_var = 'var_names',
                  time = values(list(as.POSIXct(start_model), as.POSIXct(end_model))),
                  lat = values(list(lat.min, lat.max)), lon = values(list(lon.min, lon.max)), 
                  lon_var = 'lon', lon_reorder = CircularSort(0, 360), 
                  return_vars = list(time = 'model', lon = 'model', lat = 'model'),
                  retrieve = TRUE)

rcp_data <- SeasonSelect(rcp_data, seasons = 'JJA')
```

It will be necessary an index identifying the year in which each value is recorded. In this case, a simple ordered sequence of years is built by repeating the value of the year 92 times (the number of days in summer): 

```r
years <- sort(rep(substr(start_model, 1, 4):substr(end_model, 1, 4), 92))
```

#### 2.3- Heatwaves duration


The duration of the heatwaves is obtained by using the function `Apply` with `WaveDuration` function. The parameter `case` is defined as `heat` to indicate that we are considering heatwaves and the minimum length of a heatwave is 5 days in this example. This function returns the number of days for each summer in which the maximum temperature is exceeding the 90th percentile of the reference period when they occur in a cluster of a minimum length of 5 consecutive days. This means that isolated events are not included.


```r
duration <- Apply(list(rcp_data, thresholds), target_dims = list(c('time'), c('jdays')), AtomicFun = WaveDuration, 
             case = 'heat', min.conse = 5, indices = years, output_dims = 'time')
```

The spatial representation of the maximum, mean and minimum duration of heatwaves can be plotted and saved in the working directory by running:



```r
lat <- attr(rcp_data, "Variables")$dat1$lat
lon <- attr(rcp_data, "Variables")$dat1$lon
breaks <- seq(0,92,4)
png("Spatial_Heatwave_rcp85_2020-2040_90thpercentile_1971-2000.png", width = 8, height = 12, units = 'in', 
    res = 100, type = "cairo")
par(mfrow = c(4, 1))
PlotEquiMap(apply(duration$output1, c(2, 3), max), lon = lon, lat = lat, brks = breaks, drawleg = FALSE, 
            filled.continents = FALSE, title_scale = 0.8,
            toptitle = c("Heatwave duration rcp 8.5 2020-2040 compared 90th percentile 1971-2000","Maximum"), 
            cols = heat.colors(length(breaks)-1)[(length(breaks)-1):1])
PlotEquiMap(apply(duration$output1, c(2, 3), mean), lon = lon, lat = lat, filled.continents = FALSE, brks = breaks, 
            drawleg = FALSE, toptitle = "Mean", title_scale = 0.8, 
            cols = heat.colors(length(breaks)-1)[(length(breaks)-1):1])
PlotEquiMap(apply(duration$output1, c(2, 3), min), lon = lon, lat = lat, filled.continents = FALSE, brks = breaks, 
            drawleg = FALSE, toptitle = "Minimum", title_scale = 0.8, 
            cols = heat.colors(length(breaks)-1)[(length(breaks)-1):1])
ColorBar(brks = breaks, vertical = FALSE, extra_margin = c(8,1.5,0.5,0.5), title = "Number of summer days",
            cols = heat.colors(length(breaks)-1)[(length(breaks)-1):1])
dev.off()
```

<img src="Spatial_Heatwave_rcp85_2020-2040_90thpercentile_1971-2000.png" />


The zonal average can be computed and plotted to detect differences with latitude:

```
png("Latitudinal_Heatwave_rcp85_2020-2040_90thpercentile_1971-2000.png", width = 8, height = 5, units = 'in', 
    res = 100, type = "cairo")
layout(matrix(c(1, 1, 1, 2), ncol =4, nrow=1))
matplot(2020:2040, apply(duration$output1, c(1, 2), mean), type = "l", lwd = 2, col = heat.colors(26)[26:1], 
        bty = 'n', xlab = "Time (years)", ylab = "Duration (days)", main = "Zonal Average of Heatwaves Durations", 
        cex.axis = 1.3)
ColorBar(brks = c(lat[1] - 0.9473675, lat + 0.9473675), vertical = TRUE, extra_margin = c(2, 1.5, 2, 1.5), 
         title = "Latitude", cols = heat.colors(26)[26:1])
dev.off()
```            
            
<img src = "Latitudinal_Heatwave_rcp85_2020-2040_90thpercentile_1971-2000.png" />     

#### 2.4- Heatwaves intensity

Intensities of heatwave are calculated as the maximum difference between the threshold and the maximum temperature achieved for each summer during a heatwave. Again, `Apply` function is used with `WaveIntensity` function:

```r
intensities <- Apply(list(rcp_data, thresholds), target_dims = list(c('time'), c('jdays')), AtomicFun = WaveIntensity, 
             case = 'heat', min.conse = 5, indices = years, output_dims = 'time')
```

The mean spatial distribution can be plotted and saved by running:


```r
breaks = 0:18
PlotEquiMap(apply(intensities$output1, c(2,3), mean), lon = lon, lat = lat, filled.continents = FALSE, 
            brks = breaks, drawleg = TRUE, toptitle = c("Heatwaves rcp 8.5 2020-2040 compared 90th percentile 1971-2000", 
            "Mean Intensity"), title_scale = 0.6, cols = heat.colors(length(breaks)-1)[(length(breaks)-1):1], 
            fileout = "MeanIntensityHeatwaves_rcp85_2020-2040_90thpercentile_1971-2000.png", units = "Intensity (℃)")
```

<img src = "MeanIntensityHeatwaves_rcp85_2020-2040_90thpercentile_1971-2000.png" />


The temporal evolution of the zonal average can be visualized and saved by:

```r
png("Latitudinal_Intensity_Heatwave_rcp85_2020-2040_90thpercentile_1971-2000.png", width = 8, height = 5, units = 'in', 
    res = 100, type = "cairo")
layout(matrix(c(1, 1, 1, 2), ncol = 4, nrow = 1))
matplot(2020:2040, apply(intensities$output1, c(1, 2), mean), type = "l", lwd = 2, col = heat.colors(26)[26:1], 
        bty = 'n', xlab = "Time (years)", ylab = "Intensity (℃)", main = "Zonal Average of Heatwaves Intensity", 
        cex.axis = 1.3)
ColorBar(brks = c(lat[1] - 0.9473675, lat + 0.9473675), vertical = TRUE, extra_margin = c(2, 1.5, 2, 1.5), 
          title = "Latitude", cols = heat.colors(26)[26:1])
dev.off()
```

<img src = "Latitudinal_Intensity_Heatwave_rcp85_2020-2040_90thpercentile_1971-2000.png" />


#### 2.5- Heatwaves starting date

Another interesting value can be the date in which the first day of a heatwave is recorded:

```r
start_date <- Apply(list(rcp_data, thresholds), target_dims = list(c('time'), c('jdays')), AtomicFun = WaveStart, 
             case = 'heat', min.conse = 5, indices = years, output_dims = 'time')
```


The function `WaveStart` returns the month and day in the format 'month-day'. It can be converted to a number from 1 to 92 to facilitate future calculations:
*Notice that function `WaveStart` returns 0 whether the minimum duration of the heatwaves is not reached.*

```r
summer <- c(paste(rep(6, 30), 1:30, sep = "-"), paste(rep(7, 31), 1:31, sep = "-"), paste(rep(8, 31), 1:31, sep = "-"))
start_date <- apply(start_date$output1, c(1, 2, 3), match, summer)
```

The spatial distribution of the earliest starting date of a heatwave can be plotted and saved:

```r
PlotEquiMap(apply(start_date, c(2, 3), min, na.rm = TRUE), lon = lon, lat = lat, filled.continents = FALSE, 
            fileout = "EarliestStartingDateHeatwaves_rcp85_2020-2040_90thpercentile_1971-2000.png",
            toptitle = c("Heatwaves rcp 8.5 2020-2040 compared to 90th percentile of 1971-2000", 
            "Earliest Starting Date"), title_scale = 0.6, brks = 0:92)
```

<img src = "EarliestStartingDateHeatwaves_rcp85_2020-2040_90thpercentile_1971-2000.png" />


And its temporal evolution by running:

```r
png("Latitudinal_Start_Heatwave_rcp85_2020-2040_90thpercentile_1971-2000.png", width = 8, height = 5, units = 'in', 
    res = 100, type = "cairo")
layout(matrix(c(1, 1, 1, 2), ncol = 4, nrow = 1))
matplot(2020:2040, apply(start_date, c(1, 2), mean, na.rm = TRUE), type = "l", lwd = 2, col = heat.colors(26)[26:1], 
        bty = 'n', xlab = "Time (years)", ylab = "Start Date (days)", main = "Zonal Average of Heatwaves Start Date", 
        cex.axis = 1.3)
ColorBar(brks = c(lat[1] - 0.9473675, lat + 0.9473675), vertical = TRUE, extra_margin = c(2, 1.5, 2, 1.5), 
          title = "Latitude", cols = heat.colors(26)[26:1])
dev.off()
```

<img src = "Latitudinal_Start_Heatwave_rcp85_2020-2040_90thpercentile_1971-2000.png" />


### 3- Coldwaves

An analog analysis can be done considering coldwaves. The coldwaves duration is the number of consecutive days for which the minimum temperature is below a threshold over a minimum number of days during winter.


#### 3.1- Defining coldwaves threshold

The minimum temperature during the reference period is loaded with `Start`. Notice that the starting and ending dates are modified taken into account the objective of retrieving winter data:

```r
var0 <- 'tasmin'
start_climatology <- '1970-12-01'
end_climatology <- '2000-02-28'
lat.min <- 30;  lat.max <- 80
lon.min <- -20; lon.max <- 50
climatology_filenames <- paste0("/esarchive/scratch/pbretonn/cmip5-cp4cds/historical/day/tasmin/",
                                "tasmin_day_IPSL-CM5A-LR_historical_r1i1p1_19500101-20051231.nc")

reference_data <- Start(model = climatology_filenames, var = var0, var_var = 'var_names', 
                        time = values(list(as.POSIXct(start_climatology), as.POSIXct(end_climatology))),
                        lat = values(list(lat.min, lat.max)), lon = values(list(lon.min, lon.max)), 
                        lon_var = 'lon', lon_reorder = CircularSort(0, 360), return_vars = list(time = 'model', 
                        lon = 'model', lat = 'model'), retrieve = TRUE)
metadata <- attributes(reference_data)
```

The winter data can be picked by `SeasonSelect` function from the `magic.bsc` package.

```r
reference_data <- SeasonSelect(reference_data, seasons = 'DJF')
```

To define the coldwave it is necessary to select the minimum temperature threshold that mustn't be reached. In this example, the 10 th percentile of the minimum temperature during the reference period is selected. This calculation is performed using `Threshold` function from `magic.bsc` package.

```r
quantile <- 0.1
base_range <- c(1971, 2000)
thresholds <- Threshold(reference_data, base_range = as.numeric(base_range), 
                        qtiles = quantile, ncores = detectCores() -1)
thresholds <- thresholds[c(335:365, 1:59),,]
names(dim(thresholds)) <- c('jdays', 'lat', 'lon')                    
```

#### 3.2- Coldwaves projection

The periode analized will be defined by `start_model` and `end_model` according to the objective of analyzing winter season for the period 2020-2040:


```r
start_model <- '2019-12-01'
end_model <- '2040-02-28'
model_filenames <- paste0("/esarchive/scratch/pbretonn/cmip5-cp4cds/",
                          "rcp85/day/tasmin/tasmin_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.nc")
rcp_data <- Start(model = model_filenames, var = var0, var_var = 'var_names',
                  time = values(list(as.POSIXct(start_model), as.POSIXct(end_model))),
                  lat = values(list(lat.min, lat.max)), lon = values(list(lon.min, lon.max)), 
                  lon_var = 'lon', lon_reorder = CircularSort(0, 360), 
                  return_vars = list(time = 'model', lon = 'model', lat = 'model'),
                  retrieve = TRUE)

rcp_data <- SeasonSelect(rcp_data, seasons = 'DJF')
```

An index is necessary to indicate the winter to which each value belongs, bearing in mind that December belongs to the winter of the following year. In this case, a simple ordered sequence of years is built by repeating the value of the year 90 times (the number of days in winter): 

```r
years <- sort(rep(2020:2040, 90))
```
#### 3.3- Coldwaves duration


The duration of the coldwaves is obtained by using the function `Apply` with `WaveDuration` function. The parameter `case` is defined as `cold` to indicate that we are considering coldwaves and the minimum length of a coldwave is 5 days in this example. This function returns the number of days for each winter in which the minimum temperature is below the 10th percentile of the reference period when they occur in a cluster of a minimum length of 5 consecutive days. This means that isolated events are not included.


```r
duration <- Apply(list(rcp_data, thresholds), target_dims = list(c('time'), c('jdays')), AtomicFun = WaveDuration, 
             case = 'cold', min.conse = 5, indices = years, output_dims = 'time')
```

#### 3.4- Coldwaves intensity

Intensities of coldwave are calculated as the maximum difference between the threshold and the minimum temperature achieved for each winter during a coldwave. Again, `Apply` function is used with `WaveIntensity` function:

```r
intensities <- Apply(list(rcp_data, thresholds), target_dims = list(c('time'), c('jdays')), AtomicFun = WaveIntensity, 
             case = 'cold', min.conse = 5, indices = years, output_dims = 'time')
```

#### 3.5- Coldwaves starting date

The date in wich the first winter day of a coldwave is recorded:

```r
start_date <- Apply(list(rcp_data, thresholds), target_dims = list(c('time'), c('jdays')), AtomicFun = WaveStart, 
             case = 'cold', min.conse = 5, indices = years, output_dims = 'time')
```

The function `WaveStart` returns the month and day in the format 'month-day'. It can be convert to a number from 1 to 90 to facilitate future calculations:
*Notice that function `WaveStart` returns 0 wether the minimum duration of the heatwaves is not reached.*

```r
winter <- c(paste(rep(12, 31), 1:31, sep = "-"), paste(rep(1, 31), 1:31, sep = "-"), paste(rep(2, 28), 1:28, sep = "-"))
start_date <- apply(start_date$output1, c(1, 2, 3), match, winter)
```
#### 3.6- Visualitzation of coldwaves parameters


As an example, the maximum duration projected in each gridpoint will be plotted and saved by running:

```r
breaks <- seq(0,90,4)
PlotEquiMap(apply(duration$output1, c(2, 3), max), lon = lon, lat = lat, brks = breaks, drawleg = TRUE, 
            filled.continents = FALSE, title_scale = 0.7,
            toptitle = "Coldwave maximum duration rcp 8.5 2020-2040 compared 90th percentile 1971-2000", 
            cols = heat.colors(length(breaks)-1)[(length(breaks)-1):1], 
            fileout = "Spatial_maxduration_Coldwave_rcp85_2020-2040_90thpercentile_1971-2000.png")
```


<img src = "Spatial_maxduration_Coldwave_rcp85_2020-2040_90thpercentile_1971-2000.png" />


The mean evolution of the zonal average can be simultaneously be plotted by:


```r
png("Latitudinal_Coldwave_rcp85_2020-2040_90thpercentile_1971-2000.png", width = 8, height = 8, units = 'in', 
    res = 100, type = "cairo")
layout(matrix(c(1, 2, 3, 1, 2, 3, 1, 2, 3, 4, 4, 4), ncol = 4, nrow = 3, byrow = FALSE))
matplot(2020:2040, apply(duration$output1, c(1, 2), mean), type = "l", lwd = 2, col = heat.colors(26)[26:1], 
        bty = 'n', xlab = "Time (years)", ylab = "Duration (days)", main = "Zonal Average of Coldwaves Durations", 
        cex.axis = 1.3)
matplot(2020:2040, apply(intensities$output1, c(1, 2), mean), type = "l", lwd = 2, col = heat.colors(26)[26:1], 
        bty = 'n', xlab = "Time (years)", ylab = "Intensity (℃)", main = "Zonal Average of Coldwaves Intensity", 
        cex.axis = 1.3)
  
matplot(2020:2040, apply(start_date, c(1, 2), mean, na.rm = TRUE), type = "l", lwd = 2, col = heat.colors(26)[26:1], 
        bty = 'n', xlab = "Time (years)", ylab = "Start Date (days)", main = "Zonal Average of Coldwaves Start Date", 
        cex.axis = 1.3)
ColorBar(brks = c(lat[1] - 0.9473675, lat + 0.9473675), vertical = TRUE, extra_margin = c(2, 1.5, 2, 1.5), 
         title = "Latitude", cols = heat.colors(26)[26:1])        
dev.off()
```


<img src = "Latitudinal_Coldwave_rcp85_2020-2040_90thpercentile_1971-2000.png" />
