INDECIS Quality Control Software Suite for Tall Tower wind data (INDECIS-QCSS4TT) v1.0
========================

A tall tower or met mast consists of a high vertical structure (usually reaching up to 100 meters) with several booms distributed along the vertical pole. 
If it is instrumented, meteorological measurements can be taken at different heights above ground level. 
This vignette aims to illustrate guided process through a set of Quality Control (QC) checks to ensure the high quality of tall tower wind data that has been developed 
within the context of the INDECIS project.

## 1- R packages needed 
These R libraries will be used at a some point throughout the process. Please, ensure that they are properly installed.

```r
library(data.table)
library(lubridate)
library(zoo)
library(ggplot2)
library(gridExtra)
library(scales)
```

## 2- Load configuration file containing auxiliary functions and the Quality Control software suite
```r
source('./R/config.R')
```

## 3- Preparation of the input data which will be quality controlled
Input data must be introduced in one unique data table or data frame. Column ```times``` must contain the UTC time stamps using POSIXct format (YYYY-MM-DD hh:mm:ss). Column ```freq``` must containg the time sampling interval for each observation (only these time stamp interval samplings must be considered: *"30secondly","minutely","5minutely","6minutely","10minutely","15minutely","20minutely","30minutely","hourly"*). Each of the other columns contain the meteorological variables: wind speed, wind direction and temperature for the icing test. The following **format** for each variable must be used:
- For wind speed data (in m s^-1 ): ```windagl$hgt$S$sensor_id$```
- For wind direction data (in meteorological degrees): ```wdiragl$hgt$sensor_id$```
- For temperature data (in K): ```taagl$hgt$sensor_id$```

Where ```$hgt$``` is the height above ground level where the sensor is located; ```$sensor_id$``` is a number introduced to distinguish between redundant sensors at a same height. If we do not have redundant sensors, it must be set to 1.

In addition, the data table should contain the following **attributes**:
- filtvars: variables to be quality controlled (i.e. wind speed and wind direction) + temperature variables
- heights: location of the different sensors along the mast (in meters)
- freq: time sampling interval of the original data
- sens: maximum number of sensors per height 

Load data from a tall tower located within Race Bank wind farm (UK) for the period ranging from June 2006 to December 2008.

```r
all.data <- readRDS("./data/race_bank.RData")
```

The object ```all.data``` contains wind speed, wind direction and temperature data at a several heights. The 6 first rows of the data table are shown below. Note the sensor redundancy at 88 meters for wind direction and 89 meters for wind speed.

```r
> head(all.data)
                 times wdiragl88S1 wdiragl88S2 wdiragl89S1 windagl30S1
1: 2006-06-01 00:05:00          NA       307.4          NA        4.88
2: 2006-06-01 00:15:00          NA       307.3          NA        4.61
3: 2006-06-01 00:25:00          NA       303.9          NA        4.71
4: 2006-06-01 00:35:00          NA       307.5          NA        5.05
5: 2006-06-01 00:45:00          NA       301.7          NA        4.80
6: 2006-06-01 00:55:00          NA       296.8          NA        4.17
   windagl60S1 windagl70S1 windagl80S1 windagl89S1 windagl89S2 taagl15S1
1:        4.87          NA          NA        4.84          NA    284.69
2:        4.60          NA          NA        4.60          NA    284.68
3:        4.74          NA          NA        4.69          NA    284.67
4:        5.07          NA          NA        5.06          NA    284.67
5:        4.85          NA          NA        4.85          NA    284.66
6:        4.22          NA          NA        4.33          NA    284.66
   taagl20S1 taagl88S1 taagl89S1       freq
1:    281.68        NA        NA 10minutely
2:    281.68        NA        NA 10minutely
3:    281.75        NA        NA 10minutely
4:    281.76        NA        NA 10minutely
5:    281.77        NA        NA 10minutely
6:    281.80        NA        NA 10minutely
``` 

**NOTE**: Some of the QC checks can require huge memory. Therefore, it is strongly recommended using time stamp interval samplings larger than 10 minutes. The function ```resize_time_tamps``` aggregates very high frequency data in 10-minute averages.

```r
all.data <- resize_time_stamps(all.data)
```

## 4- Run the INDECIS-QCSS4TT

A total of 15 QC routines have been considered and coded to ensure the high quality of tall tower wind data. In addition, 2 more QC checks could be performed preliminary, but they have not been coded here. These two tests are:
- **Surroundings check**: it is highly recommended to characterise the area where the tower is placed. For example, a high canopy around the mast could increase significantly the surface roughness and reduce wind speeds especially near the ground level. 
- **Time stamps check**: this test must detect all the inconsistencies in the time stamps produced by errors in the data logger. 

The output of each QC test is a data table. This object is the same as the input data table, but the routine will append one flag column for each test that has been run and variable that has been checked. Added columns will be named using the pattern ```flag$qc$_$var$```, where ```$qc$``` refers to the QC routine name and ```$var$``` is the variable name which has been checked. 
Note that any observation will be removed or set to NA after running the INDECIS-QSS4TT.

The user can decide the order of the checks as well as consider which tests would like to run or not.

Flag columns contain a whole number indicating whether an observation passes the QC check successfully or not. Five different categories have been defined by imposing several levels of confidence using specific thresholds. These values are based on the [**WMO standards**] (http://www.wmo.int/pages/prog/www/OSY/Manual/488_Guide_2007.pdf) and after running the QC routines over wind data from more than 200 tall towers. QC flags are:

- **0**: the observation has not been evaluated
- **1**: the observation has passed the test successfully
- **2**: the observation has passed the test, but could need further check
- **4**: the observation fails to pass the test successfully
- **5**: calm winds. Wind speeds below 0.5 m s^-1 are not considered in the majority of the tests.
- **9**: missing observation

### 4.1- Plausible values check

This test detects and flags unrealistic values. Negative wind speed values or observations above ```max.value.fail``` are flagged as 'Fail'. Wind direction values falling outside the range [0,360] are flagged as 'Fail'. Wind speed values within the interval (```max.value.suspect```,```max.value.fail```) are considered as 'Suspect'. By default, ```max.value.suspect``` and ```max.value.fail``` are set to 75 m s^-1 (Vaisala's sensors highest recordable value)
and 113.2 m s^-1 (wind gust world record), respectively. Threshold values can be changed by modifying the function arguments  ```max.value.suspect``` and ```max.value.suspect```.

```r
all.data <- plausible_values(all.data)
```

### 4.2- Difference between extreme values check

Detect and flag untrue extreme wind speed values of the time series by checking the difference between the maximum and the second maximum values of the empirical distribution.
If the difference betweeen the first maximum and the second maximum exceeds the absolute value of the second maximum, the first maximum is flagged as 'Suspect'. This test is runs iteratively.
```r
all.data <- diff_extreme_values(all.data)
```

### 4.3- Step test

Detect and flag those pairs of consecutive wind speed values which difference exceeds a certain threshold. This threshold depends on the time stamp interval sampling. 
It is set to 20 m s⁻1 for 2-minutely or higher resolution data. For higher time stamp intervals, the threshold is set to 10 m s^-1 ([Jimenez et al., 2010] (https://journals.ametsoc.org/doi/full/10.1175/2010JTECHA1404.1)). 

```r
all.data <- step_test(all.data)
```

### 4.4- Rate of change test

Uses the same methodology as the 'Step test'. However, the threshold depends on the statistical distribution of the time series. 
If The difference between two consecutive observations is higher than 3·IQR, both values are flagged as 'Fail'. If this difference falls between 2·IQR and 3·IQR, 
the pair of values are flagged as 'Suspect'.

```r
all.data <- rate_change(all.data)
```

### 4.5- Abnormal variations

**NOTE**: This test might require large amount of RAM, specially for high resolution data.

This check detects inadequate variations (abnormally high or abnormally low) within the time series. It computes 30-day moving variances over wind speed time series. 
If the standard deviation of a specific 30-day period departs more than 4 standard deviations of the mean standard deviation of all the 30-day periods within the time series are flagged as 'Suspect'.

```r
all.data <- abnormal_variations(all.data)
```

### 4.6- Systematic errors

**NOTE**: This test might require large amount of RAM, specially for high resolution data.

This check detects random errors in the experimental measurements. Systematic errors test computes 30-day moving averages over wind speed time series. 
If the mean wind speed of a specific 30-day period departs more than 4 standard deviations of the mean value of all 30-day period means within the time series are flagged as 'Suspect'.

```r
all.data <- systematic_errors(all.data)
```

### 4.7- Quartile occurrences check

This test detects random errors in the experimental measurements. It looks for the frequency of appearance of the first, second and third quartiles in the wind speed time series. 
The table below summarises the different thresholds (in days) that define the success flags (i.e. 'Pass', 'Suspect' and 'Fail'). As an example, the first row implies that if all the observations 
in a 30-day period are above the first quartile of the distribution, data from this time period will be flagged as 'Fail'. The observation is flagged as 'Suspect' when
the period without any appearance  of the first quartile ranges between 15 and 30 days. It is considered 'Pass' when the duration is less than 15 days.

| **All the observations are ...** | **Pass** | **Suspect** | **Fail** |
| :------------------------------: | :------: | :---------: | :------: |
| > 1^st quartile                  | < 15     | [15,30]     | > 30     |
| > 2^nd quartile                  | < 10     | [10,20]     | > 20     |
| > 3^rd quartile                  | < 5      | [5,10]      | > 10     |
| < 1^st quartile                  | < 5      | [5,10]      | > 10     |
| < 2^nd quartile                  | < 10     | [10,20]     | > 20     |
| < 3^rd quartile                  | < 15     | [15,30]     | > 30     |

```r
all.data <- quartile_occurrences(all.data)
```

### 4.8- Persistence test

Detect and flag sequences of wind speed and wind direction observations with extremely low variability. 

For very high resolution observations (i.e. 1-minutely or more) data are flagged as 'Suspect' when:
- Wind speed does not change more than 0.5 m s^-1 in 60 consecutive values in a row.
- Wind direction does not change more than 10 degrees in 60 consecutive values.

For time stamp samplings between 1 minute and 1 hour, data are flagged as 'Suspect' when:
- Wind speed does not change more than 0.7 m s -1 in 60 consecutive values.
- Wind direction does not change more than 5 degrees in 60 consecutive values.

It is important to point out that relatively long periods with very low variability where the mean wind speed value is close to zero are 
typical of the natural observed variability. Hence, these data should be considered reliable. Persistence test does not introduce any success flag to observations 
lower than 0.5 m s^-1. These measurements are then flagged as 'Calm'.  

```r
all.data <- persistence_test(all.data)
```

### 4.9- Flat line test

Identify and flag sequences of constant wind speed and wind direction observations.

Data are flagged as 'Fail' when the test detects:
- 6 or more constant wind speed observations in a row or
- periods of 30 days in which the maximum wind speed value is 0 or
- 40 or more constant wind direction observations in a row.

Observations are flagged as 'Suspect' when the check identifies:
- 3 or more constant wind speed observations in a row or
- 40 or more constant wind direction observations in a row.

```r
all.data <- flat_line(all.data)
```

### 4.10- Icing test

**NOTE**: This test needs temperature observations either measured at surface level or at several heights along the tower. 

Detect and flag icing periods when an anemometer has been frosted and produces erroneous measurements. 
Data are flagged as 'Fail' when the test detects 4 or more days with 0 m s^-1 as the maximum wind speed value and below zero temperatures. 

```r
all.data <- icing(all.data)
```

<img src="icing_hegyhatsal.png" />

### 4.11- Repeated sequences test

Detect and flag sequences of wind speed and wind direction observations that appear repeated within the same time series. These sequences are flagged as
'Fail' when they contain:

- 20 or 30 (depending on the time stamp sampling) wind speed values
- 30 wind direction observations

```r
all.data <- repeated_sequences(all.data)
```

**EXAMPLE**: The image below shows an example of a detection of a repeated sequence in a same time series in three different measurement levels. The sequence of wind 
speed values belonging to the period ranging from December 22 to December 28 are all equal to the wind speeds from December 30 to January 4.
 
<img src="repeated_sequence_abadan.png" />

### 4.12- Tower shadows

Look for wind directions that are in the wake produced by the vertical structure of the mast. Wind speed values falling in the wake directions are flagged as 'Suspect'.
These directions are identified by computing the ratio between wind speeds measured by redundant sensors. Then, this wind speed ratios are grouped by wind direction. 
Wind directions which ratios fall below the 5th percentile and above 95th percentile of the ratios distribution are flagged as 'Suspect'.

```r
all.data <- tower_shadows(all.data)
```

<img src="tower_shadow_fino3.png" />

### 4.13- Vertical ratios

Detect and flag erroneous measurements by comparing simultaneous wind speed observations at different tower measurement levels. 
The test computes the ratios between pairs of simultaneous measurements. The pair of values is flagged as 'Fail' if the quotient between them departs more than 
30 points from the mean value of the distribution generated by all the pair-wise observations quotients. It it departs between 15 and 30 points, both values are flagged 
'Suspect'.

```r
all.data <- vertical_ratios(all.data)
```

### 4.14- Isolated pass

**NOTE**: This test must be applied after running at least one of the other routines. 

Detect and re-flag values that have been previously flagged as 'Pass' by some of the tests, but appear bounded by suspicious or wrong observations. Specific flag 
sequences have been defined and the code looks for them in the ```flag$qc$_$var$``` columns.

'Pass' and 'Calm' flags falling in the middle of these sequences are changed into 'Fail':

```r
c(rep("Fail",3),rep("Pass",1),rep("Fail",3))
c(rep("Fail",5),rep("Pass",2),rep("Fail",5))
c(rep("Fail",10),rep("Pass",3),rep("Fail",10))
c(rep("Fail",15),rep("Pass",4),rep("Fail",15))
c(rep("Fail",25),rep("Pass",5),rep("Fail",25))
c(rep("Missing",50),rep("Pass",1),rep("Missing",50))
c(rep("Missing",50),rep("Calm",1),rep("Missing",50)) 
```

'Pass' flags falling in the middle of these sequences are changed into 'Suspect':

```r
c(rep("Suspect",3),rep("Pass",1),rep("Suspect",3)) 
c(rep("Suspect",5),rep("Pass",2),rep("Suspect",5))
c(rep("Suspect",10),rep("Pass",3),rep("Suspect",10))
c(rep("Suspect",15),rep("Pass",4),rep("Suspect",15))
c(rep("Suspect",25),rep("Pass",5),rep("Suspect",25))
```

```r
all.data <- isolated_pass(all.data)
```

### 4.15- Occurrences of 0 and 360 values

This is a QC check of the whole time series. Thus, any single observation is flagged. Instead of this, two attributes (```perc.of.zeros``` and ```perc.of.360```) 
are added to the data table indicating the percentage of occurrences of 0s within the wind speed and wind direction time series and 360s for wind direction 
time series with respect to all the observations, respectively. High percentages of these values can mirror the existence of erroneous measurements produced by 
internal conventions from the data provider that have not been taken into account.

```r
all.data <- zerosand360(all.data)
```

### 4.16- Internal consistency check

**NOTE**: This check must be applied after defining a convention and introducing it in the wind time series.

Ensure that a specific convention -previously defined- has been properly applied to wind speed and wind direction time series. Pairs of observations that do not match
the convention: 

```wind speed == ws & wind direction == wd``` 

will be flagged as 'Fail'. Function parameters ```ws``` and ```wd``` can be changed. By default, they are set to ```0``` and ```NA``` respectively. 

```r
all.data <- internal_consistency(all.data)
```