INDECIS Quality Control Software Suite for Tall Tower wind data (INDECIS-QCSS4TT) v1.0
========================

A tall tower or met mast consists of a high vertical structure (often reaching heights up to 100 meters above ground) with several booms distributed along the vertical mast. 
Those booms are instrumented with meteorological sensors that record wind speed, wind direction and other parameters at multiple heights above ground level. 
The special nature of this observational sites allows to perfrom some non-standard quality control techniques.
This vignette illustrates step by step the process of Quality Controlling (QC) wind speed observations from tall towers using the functions that have been developed 
within the context of the INDECIS project.

## 1- R packages needed 
The INDECIS-QCSS4TT functions require some R libraries to be installed. Those packages can be installed from CRAN archive:

```r
install.packages("data.table","lubridate","zoo","ggplot2","gridExtra","scales")
```

## 2- Load configuration file containing auxiliary functions and the Quality Control software suite
All the QC functions need to be loaded before starting. This is achieved by sourcing a single configuration file that takes care of loading all the functions:

```r
source('./R/config.R')
```

## 3- Preparation of the input data which will be quality controlled
The INDECIS-QCSS4TT requires all the tall tower data to be in a specific format. Therefore input data needs to be adjusted to this format. 
All data from each tall tower must be introduced in a single ```data.table``` or ```data.frame``` object containing:
- ```times``` column must list the UTC time stamps as a POSIXct type. 
- ```freq```  column must contain the time sampling interval for each observation. Only these samplings are 
allowed: *"30secondly","minutely","5minutely","6minutely","10minutely","15minutely","20minutely","30minutely","hourly"*. Note that the time stamp sampling can change
along the time series, either because of a sensor replacement or change in the measurement sampling rate.
- Each of the other columns contain the meteorological variables: wind speed, wind direction and temperature (employed for icing tests). 

As for each tower there can exist several sensors at different booms and heights, this infromation needs to be contained in the column name. 
The following **conventions** have to be used:
- For wind speed data (in m s^-1 ): ```windagl[hgt]S[sensor_id]```
- For wind direction data (in meteorological degrees): ```wdiragl[hgt]S[sensor_id]```
- For temperature data (in K): ```taagl[hgt]S[sensor_id]```

Where ```[hgt]``` has to be replaced by the sensor height above ground level; ```[sensor_id]``` is a number that uniquely identifies redundant sensors at a 
same height (i.e. more than one sensor at a certain measurement level). It must be set to 1 for non-redunded sensors.

In addition, the data table should contain the following **attributes**:
- filtvars: variables to be quality controlled (i.e. wind speed and wind direction) + temperature variables
- heights: location of the different sensors along the mast (in meters)
- freq: time sampling interval of the original data. 
- sens: maximum number of sensors per height 

For demonstration purposes, we load data from a tall tower located within Race Bank wind farm (UK) for the period ranging from June 2006 to December 2008.

```r
all.data <- readRDS("./data/race_bank.RData")
```

The object ```all.data``` contains wind speed, wind direction and temperature observations at a several heights. The first six rows of the data.table are shown below. Note the sensor redundancy at 88 meters for wind direction and 89 meters for wind speed.

```r
> head(all.data)
                 times wdiragl88S1 wdiragl88S2 wdiragl89S1 windagl30S1 windagl60S1 windagl70S1 windagl80S1 windagl89S1 windagl89S2 taagl15S1 taagl20S1 taagl88S1 taagl89S1       freq

1: 2006-06-01 00:05:00          NA       307.4          NA        4.88        4.87          NA          NA        4.84          NA    284.69     281.68        NA        NA 10minutely
2: 2006-06-01 00:15:00          NA       307.3          NA        4.61        4.60          NA          NA        4.60          NA    284.68     281.68        NA        NA 10minutely
3: 2006-06-01 00:25:00          NA       303.9          NA        4.71        4.74          NA          NA        4.69          NA    284.67     281.75        NA        NA 10minutely
4: 2006-06-01 00:35:00          NA       307.5          NA        5.05        5.07          NA          NA        5.06          NA    284.67     281.76        NA        NA 10minutely
5: 2006-06-01 00:45:00          NA       301.7          NA        4.80        4.85          NA          NA        4.85          NA    284.66     281.77        NA        NA 10minutely
6: 2006-06-01 00:55:00          NA       296.8          NA        4.17        4.22          NA          NA        4.33          NA    284.66     281.80        NA        NA 10minutely
``` 

And the corresponding attributes using the format mentioned above are:

```r
> attr(all.data,"filtvars")
 [1] "wdiragl88S1" "wdiragl88S2" "wdiragl89S1" "windagl30S1" "windagl60S1" "windagl70S1" "windagl80S1" "windagl89S1" "windagl89S2" "taagl15S1"  "taagl20S1"   "taagl88S1"   "taagl89S1"  
> attr(all.data,"heights")
$windagl
[1] 30 60 70 80 89

$wdiragl
[1] 88 89

$taagl
[1] 15 20 88 89
> attr(all.data,"freq")
[1] "10minutely"
> attr(all.data,"sens")
$windagl
[1] 2

$wdiragl
[1] 2

$taagl
[1] 1
```

Two preliminary checks should be carried out before introducing the data into the INDECIS-QCSS4TT routines. These tests are not programmed here, as they cannot be either coded
or depend on the original data collection and formatting processes.

- **Surroundings check**: it is highly recommended to characterise the area where the tower is placed. For example, a high canopy around the mast could increase significantly the surface roughness and reduce wind speeds especially near the ground level. Changes in the tower environment can produce trends in the records.
- **Time stamps check**: ensure that all the time stamps appear in the input data table and equally spaced according to the time stamp samplings specified in the attribute ```freq```. 

**NOTE**: Some of the QC checks have huge memory requirements. Therefore, it is strongly recommended to use a sampling interval of 10 minutes or larger. The function ```resize_time_tamps``` aggregates higher frequency data to 10-minute averages.

```r
all.data <- resize_time_stamps(all.data)
```

## 4- Run the INDECIS-QCSS4TT

A total of 16 QC routines have been added to the previos mentioned preliminray checks and coded to detect problems in tall tower wind data. A flux diagram of the QC tests is shown in the flowchart below. 
In except of the *Isolated pass* test, the QC routines can be run independently so that any user of this software can redefine the order and decide whether 
a test is applied or not. 

<div style="text-align:center"><img src="./img/QC_flux_diagram.png" / width="520" height="920"></div>

The output of each QC test is a new data.table with the same columns and data as the input data.table, but the routine will append one flag column for each test that has been run and variable that has been checked. Added columns will be named using the pattern ```flag[qc]_[var]```, where ```[qc]``` refers to the QC routine name and ```[var]``` is the variable name which has been checked. 
Note that no observation will be removed or set to NA by any of the INDECIS-QSS4TT tests.

The user can decide the order of the checks that wants to perform as well as to exclude any of them.

Flag columns contain a whole number indicating whether an observation passes the QC check successfully or not. In this way, three different categories have been 
defined depending on whether an observation passes the test successfully (*Pass*); passes the test but could need further check (*Suspect*); or fails the test (*Fail*).
These classification has been done by setting different threshold values based on the World Meteorological Organization standards (WM0, 2007), manuals (Brower, 2012; 
IEC, 2005 and QARTOD, 2017) or scientific articles (Jimenez et al., 2010) and after testing them over observations from more than 200 tall towers. In addition, 
three more flags have been introduced to indicate if an observation has not been evaluated by the test, is missing, or corresponds to calm winds.

- **0**: the observation has not been evaluated
- **1**: the observation has passed the test successfully
- **2**: the observation has passed the test, but could need further check
- **4**: the observation fails to pass the test successfully
- **5**: calm winds. Wind speeds below 0.5 m s^-1 are not considered in the majority of the tests.
- **9**: missing observation

### 4.1- Plausible values check

This test detects and flags unrealistic values. Negative wind speed values or observations above ```max.value.fail``` are flagged as 'Fail'. Wind direction values falling outside the range [0,360] are flagged as 'Fail'. Wind speed values within the interval (```max.value.suspect```,```max.value.fail```) are considered as 'Suspect'. By default, ```max.value.suspect``` and ```max.value.fail``` are set to 75 m s^-1 (Vaisala's sensors highest recordable value)
and 113.2 m s^-1 (wind gust world record), respectively. Threshold values can be changed by modifying the function arguments  ```max.value.suspect``` and ```max.value.suspect```.

```r
all.data <- plausible_values(all.data)
```

### 4.2- Difference between extreme values check

Detect and flag untrue extreme wind speed values of the time series by checking the difference between the maximum and the second maximum values of the empirical distribution.
If the difference between them exceeds the absolute value of the second maximum, the first maximum is flagged as 'Suspect'. This test is runs iteratively.
```r
all.data <- diff_extreme_values(all.data)
```

### 4.3- Persistence test

Detect and flag sequences of wind speed and wind direction observations with extremely low variability. 

For very high resolution observations (i.e. 1-minutely or more) data are flagged as 'Suspect' when:
- Wind speed does not change more than 0.5 m s^-1 in 60 consecutive values in a row.
- Wind direction does not change more than 10 degrees in 60 consecutive values.

For time stamp samplings between 1 minute and 1 hour, data are flagged as 'Suspect' when:
- Wind speed does not change more than 0.7 m s -1 in 60 consecutive values.
- Wind direction does not change more than 5 degrees in 60 consecutive values.

It is important to point out that relatively long periods with very low variability where the mean wind speed value is close to zero are 
typical of the natural observed variability. Hence, these data should be considered reliable. Persistence test does not introduce any success flag to observations 
lower than 0.5 m s^-1. These measurements are then flagged as 'Calm'.  

```r
all.data <- persistence_test(all.data)
```

### 4.4- Flat line test

Identify and flag sequences of constant wind speed and wind direction observations.

Data are flagged as 'Fail' when the test detects:
- 6 or more constant wind speed observations in a row or
- periods of 30 days in which the maximum wind speed value is 0 or
- 40 or more constant wind direction observations in a row.

Observations are flagged as 'Suspect' when the check identifies:
- 3 or more constant wind speed observations in a row or
- 20 or more constant wind direction observations in a row.

```r
all.data <- flat_line(all.data)
```

### 4.5- Icing test

**NOTE**: This test needs temperature observations either measured at surface level or at several heights along the tower. 

Detect and flag icing periods when an anemometer has been frosted and produces erroneous measurements. 
Data are flagged as 'Fail' when the test detects 4 or more days with 0 m s^-1 as the maximum wind speed value and below zero temperatures. 

```r
all.data <- icing(all.data)
```

**EXAMPLE**: An Icing test detected in Hegyhátsál tall tower (Hungary) is shown in the plot below. Note the zero wind speed records during 10 days in-a-row (from 
December 8 to December 18) for the two uppermost levels. Temperatures were below zero during all the period (not shown). 

<img src="./img/icing_hegyhatsal.png" />

### 4.6- Abnormal variations

**NOTE**: This test might require large amount of RAM, specially for high resolution data.

This check detects inadequate variations (abnormally high or abnormally low) within the time series. It computes 30-day moving variances over wind speed time series. 
If the standard deviation of a specific 30-day period departs more than 4 standard deviations of the mean standard deviation of all the 30-day periods within the time series are flagged as 'Suspect'.

```r
all.data <- abnormal_variations(all.data)
```

### 4.7- Systematic errors

**NOTE**: This test might require large amount of RAM, specially for high resolution data.

This check detects random errors in the experimental measurements. Systematic errors test computes 30-day moving averages over wind speed time series. 
If the mean wind speed of a specific 30-day period departs more than 4 standard deviations of the mean value of all 30-day period means within the time series are flagged as 'Suspect'.

```r
all.data <- systematic_errors(all.data)
```

### 4.8- Quartile occurrences check

**NOTE**: It is strongly recommended the user removes unrealistic values and flat line sequences to carry out the quartiles calculation. Therefore, the user should run first
the 'Plausible values' and 'Flat line' tests.

This test detects random errors in the experimental measurements. It looks for the frequency of appearance of the first, second and third quartiles in the wind speed time series. 
The table below summarises the different thresholds (in days) that define the success flags (i.e. 'Pass', 'Suspect' and 'Fail'). As an example, the first row implies that if all the observations 
in a 30-day period are above the first quartile of the distribution, data from this time period will be flagged as 'Fail'. The observation is flagged as 'Suspect' when
the period without any appearance  of the first quartile ranges between 15 and 30 days. It is considered 'Pass' when the duration is less than 15 days.

| **All the observations are ...** | **Pass** | **Suspect** | **Fail** |
| :------------------------------: | :------: | :---------: | :------: |
| > 1^st quartile                  | < 15     | [15,30]     | > 30     |
| > 2^nd quartile                  | < 10     | [10,20]     | > 20     |
| > 3^rd quartile                  | < 5      | [5,10]      | > 10     |
| < 1^st quartile                  | < 5      | [5,10]      | > 10     |
| < 2^nd quartile                  | < 10     | [10,20]     | > 20     |
| < 3^rd quartile                  | < 15     | [15,30]     | > 30     |

```r
all.data <- quartile_occurrences(all.data)
```

### 4.9- Rate of change test

Uses the same methodology as the 'Step test'. However, the threshold depends on the statistical distribution of the time series. 
If The difference between two consecutive observations is higher than 3·IQR, both values are flagged as 'Fail'. If this difference falls between 2·IQR and 3·IQR, 
the pair of values are flagged as 'Suspect'.

```r
all.data <- rate_change(all.data)
```

### 4.10- Step test

Detect and flag those pairs of consecutive wind speed values which difference exceeds a certain threshold. This threshold depends on the time stamp interval sampling. 
It is set to 20 m s⁻1 for 2-minutely or higher resolution data. For higher time stamp intervals, the threshold is set to 10 m s^-1 ([Jimenez et al., 2010] (https://journals.ametsoc.org/doi/full/10.1175/2010JTECHA1404.1)). 

```r
all.data <- step_test(all.data)
```

### 4.11- Repeated sequences test

Detect and flag sequences of wind speed and wind direction observations that appear repeated within the same time series. These sequences are flagged as
'Fail' when they contain:

- 20 or 30 (depending on the time stamp sampling) wind speed values
- 30 wind direction observations

```r
all.data <- repeated_sequences(all.data)
```

**EXAMPLE**: The image below shows an example of a detection of a repeated sequence within a same time series in Abadan tall tower (Iran). In addition, repeated 
sequences have been found in all three different measurement levels. Note that the sequence of wind speed values belonging to the period ranging from December 22 
to December 28 are all equal to the wind speeds from December 30 to January 4.
 
<img src="./img/repeated_sequence_abadan.png" />

### 4.12- Tower shadow

Look for wind directions that are in the wake produced by the vertical structure of the mast. Wind speed values falling in the wake directions are flagged as 'Suspect'.
These directions are identified by computing the ratio between wind speeds measured by redundant sensors. Then, this wind speed ratios are grouped by wind direction. 
Wind directions which ratios fall below the 5th percentile and above 95th percentile of the ratios distribution are flagged as 'Suspect'.

```r
all.data <- tower_shadow(all.data)
```

**EXAMPLE**: The plot below represents the quotient between wind speeds measured by two redundant sensors (hereafter referred as S1 and S2) from an imaginary tall tower. 
These wind speeds ratios have been grouped by wind direction given by the closest windvane within the tower. One can note an increase of the ratio S1/S2 between 40º and
60º, showing values higher than 1.1. This implies that wind speeds measured by S1 are systematically stronger than those observed by S2. This reduction of wind speeds (shadow) 
over S2 are produced by the vertical structure of the mast. Therefore, wind speeds measured by S2 coming from directions between 40º and 60º are not reliable and the 
'tower shadows' check will flag them as 'Suspect'. An analogous situation is observed for S1 for wind speeds coming from directions between 160º and 180º.

<img src="./img/tower_shadow_fino3.png" />

### 4.13- Vertical ratios

Detect and flag erroneous measurements by comparing simultaneous wind speed observations at different tower measurement levels. 
The test computes the ratios between pairs of simultaneous measurements. The pair of values is flagged as 'Fail' if the quotient between them departs more than 
30 points from the mean value of the distribution generated by all the pair-wise observations quotients. It it departs between 15 and 30 points, both values are flagged 
'Suspect'.

```r
all.data <- vertical_ratios(all.data)
```

### 4.14- Isolated pass

**NOTE**: This test must be applied after running at least one of the other routines. 

Detect and re-flag values that have been previously flagged as 'Pass' by some of the tests, but appear bounded by suspicious or wrong observations. Specific flag 
sequences have been defined and the code looks for them in the ```flag$qc$_$var$``` columns.

'Pass' and 'Calm' flags falling in the middle of these sequences are changed into 'Fail':

```r
c(rep("Fail",3),rep("Pass",1),rep("Fail",3))
c(rep("Fail",5),rep("Pass",2),rep("Fail",5))
c(rep("Fail",10),rep("Pass",3),rep("Fail",10))
c(rep("Fail",15),rep("Pass",4),rep("Fail",15))
c(rep("Fail",25),rep("Pass",5),rep("Fail",25))
c(rep("Missing",50),rep("Pass",1),rep("Missing",50))
c(rep("Missing",50),rep("Calm",1),rep("Missing",50)) 
```

'Pass' flags falling in the middle of these sequences are changed into 'Suspect':

```r
c(rep("Suspect",3),rep("Pass",1),rep("Suspect",3)) 
c(rep("Suspect",5),rep("Pass",2),rep("Suspect",5))
c(rep("Suspect",10),rep("Pass",3),rep("Suspect",10))
c(rep("Suspect",15),rep("Pass",4),rep("Suspect",15))
c(rep("Suspect",25),rep("Pass",5),rep("Suspect",25))
```

```r
all.data <- isolated_pass(all.data)
```

### 4.15- Occurrences of 0 and 360 values

This is a QC check of the whole time series. Thus, any single observation is flagged. Instead of this, two attributes (```perc.of.zeros``` and ```perc.of.360```) 
are added to the data table indicating the percentage of occurrences of 0s within the wind speed and wind direction time series and 360s for wind direction 
time series with respect to all the observations, respectively. High percentages of these values can mirror the existence of erroneous measurements produced by 
internal conventions from the data provider that have not been taken into account.

```r
all.data <- zerosand360(all.data)
```

### 4.16- Internal consistency check

**NOTE**: This check must be applied after defining a convention and introducing it in the wind time series.

Ensure that a specific convention -previously defined- has been properly applied to wind speed and wind direction time series. Pairs of observations that do not match
the convention: 

```wind speed == ws & wind direction == wd``` 

will be flagged as 'Fail'. Function parameters ```ws``` and ```wd``` can be changed. By default, they are set to ```0``` and ```NA``` respectively. 

```r
all.data <- internal_consistency(all.data)
```

## References
Brower M.C. A practical guide to developing a wind project. 2012, John Wiley and Sons

IEC 61400-12-1. ed. 1.0. Geneva, Switzerland: IEC. Copyright © 2005.

Jimenez P.A., González-Rouco J.L., Navarro J., Montávez J.P., García-Bustamante E. Quality Assurance of Surface Wind Observations from Automated Weather Stations, 2010. Journal of Atmospheric and Oceanic Technology, 27: pp. 1101-1122.

QARTOD, IOOS. Manual for Real-Time Quality Control of Wind Data A Guide to Quality Control and Quality Assurance for Coastal and Oceanic Wind Observations. Version 1.1, 2017.

WMO. Guidelines on Quality Control Procedures for Data from Automatic Weather Stations. Third Edition. Geneva, Switzerland, 2007.