Skip to content
GitLab

Home · Changes

Page history
Update home authored by vagudets's avatar vagudets
Hide whitespace changes
Inline Side-by-side
home.md
View page @ 54b4a249
## Introduction ## Introduction
SUNSET is a modular tool for subseasonal to seasonal to decadal forecast verification workflows. It is intended to have a modularized structure, where each module is a separate part of the code that performs a specific task, so that parts of the workflow can be skipped or reordered. This the GitLab repository for the SUNSET (SUbseasoNal to decadal climate forecast post-processIng and asSEmenT suite), a modular tool for subseasonal to seasonal to decadal forecast verification workflows. It is intended to have a modularized structure, where each module is a separate part of the code that performs a specific task, so that parts of the workflow can be skipped or reordered.
The datasets, forecast horizon, time period, skill metrics to compute and other parameters are specified by the user in a configuration file, called "recipe". The datasets, forecast horizon, time period, skill metrics to compute and other parameters are specified by the user in a configuration file, called "recipe".
After definining the recipe, the user can create a script using the functions available in the tool, which use apply the information in the recipe to do the desired computations. You an example script [in the SUNSET code Snippets](https://earth.bsc.es/gitlab/es/sunset/-/snippets/111). After definining the recipe, the user can create a script using the functions available in the tool, which use apply the information in the recipe to do the desired computations. You an example script [in the SUNSET code Snippets](https://earth.bsc.es/gitlab/es/sunset/-/snippets/111).
- Modules currently available: Loading, Calibration, Anomalies, Skill, Saving, Visualization - Modules currently available: Loading, Calibration, Anomalies, Skill, Indices, Saving, Visualization
- Modules in development: Downscaling, Scorecards - Modules in development: Downscaling, Scorecards
- Future modules: Aggregation, Indicators - Future modules: Aggregation, Indicators
This tool is in the early stages of development, so the code and the information in this wiki may be subject to frequent changes and updates. This wiki contains all the information needed to use the available modules. This tool is in the early stages of development, so the code and the information in this wiki may be subject to frequent changes and updates. This wiki contains all the information needed to use the available modules.
## Recipes ## Recipes
In order to use the Verification Suite, users must define a recipe containing all the information pertaining to their workflow. In order to use SUNSET, users must define a recipe containing all the information pertaining to their workflow.
Here is an example of a recipe to load monthly mean ECMWF System 5 data from `/esarchive/`, with a 1993 to 2016 hindcast period, the corresponding ERA5 observations, and a 2020 forecast for the November initialization, for the months of November and December. Here is an example of a recipe to load monthly mean ECMWF System 5 data from `/esarchive/`, with a 1993 to 2016 hindcast period, the corresponding ERA5 observations, and a 2020 forecast for the November initialization, for the months of November and December.
The observations will be interpolated to the experiment grid (Regrid type: 'to_system') using bilinear interpolation. The hindcast and forecast will be calibrated using Quantile Mapping, and the Ranked Probability Skill Score (RPSS) and Continuous Ranked Probability Skill Score (CRPSS) will be computed.
Following this recipe, the observations will be interpolated to the experiment grid (Regrid type: 'to_system') using bilinear interpolation. The hindcast and forecast will be calibrated using Quantile Mapping, and the Ranked Probability Skill Score (RPSS) and Continuous Ranked Probability Skill Score (CRPSS) will be computed.
The terciles (1/3, 2/3), quartiles (1/4, 2/4, 3/4), extremes (1/10, 9/10) and their corresponding probability bins will also be computed. Any output files will be saved inside the output directory. The terciles (1/3, 2/3), quartiles (1/4, 2/4, 3/4), extremes (1/10, 9/10) and their corresponding probability bins will also be computed. Any output files will be saved inside the output directory.
```yaml ```yaml
...@@ -30,54 +30,92 @@ Description: ...@@ -30,54 +30,92 @@ Description:
Analysis: Analysis:
Horizon: seasonal # Mandatory, str: 'subseasonal', 'seasonal', or 'decadal' Horizon: seasonal # Mandatory, str: 'subseasonal', 'seasonal', or 'decadal'
Variables: Variables:
name: tas # Mandatory, str: variable name in /esarchive/ # name: variable name(s) in the /esarchive (Mandatory, str)
freq: monthly_mean # Mandatory, str: 'monthly_mean' or 'daily_mean' # freq: 'monthly_mean', 'daily' or 'daily_mean' (Mandatory, str)
- {name: 'tas', freq: 'monthly_mean', units: 'C'}
Datasets: Datasets:
System: System:
name: ECMWF-SEAS5 # Mandatory, str: System name. # name: System name (Mandatory, str)
Multimodel: no # Mandatory, bool: Either yes/true or no/false # member: 'all' or individual members, separated by a comma and in quotes (decadal only, str)
- {name: 'ECMWF-SEAS5', member: 'all'}
Multimodel: no # Either yes/true or no/false (Mandatory, bool)
Reference: Reference:
name: ERA5 # Mandatory, str: Reference name. - {name: 'ERA5'} # Reference name (Mandatory, str)
Time: Time:
sdate: '1101' # Mandatory, int: Start date, 'mmdd' sdate: '1101' # Start date, 'mmdd' (Mandatory, int)
fcst_year: '2020' # Optional, int: Forecast initialization year 'YYYY' fcst_year: '2020' # Forecast initialization year 'YYYY' (Optional, int)
hcst_start: '1993' # Mandatory, int: Hindcast initialization start year 'YYYY' hcst_start: '1993' # Hindcast initialization start year 'YYYY' (Mandatory, int)
hcst_end: '2016' # Mandatory, int: Hindcast initialization end year 'YYYY' hcst_end: '2016' # Hindcast initialization end year 'YYYY' (Mandatory, int)
ftime_min: 1 # Mandatory, int: First forecast time step in months. Starts at “1”. ftime_min: 1 # First forecast time step in months. Starts at “1”. (Mandatory, int)
ftime_max: 6 # Mandatory, int: Last forecast time step in months. Starts at “1”. ftime_max: 6 # Last forecast time step in months. Starts at “1”. (Mandatory, int)
Region: Region:
latmin: -10 # Mandatory, int: minimum latitude latmin: -90 # minimum latitude (Mandatory, int)
latmax: 10 # Mandatory, int: maximum latitude latmax: 90 # maximum latitude (Mandatory, int)
lonmin: 0 # Mandatory, int: minimum longitude lonmin: 0 # minimum longitude (Mandatory, int)
lonmax: 20 # Mandatory, int: maximum longitude lonmax: 359.9 # maximum longitude (Mandatory, int)
Regrid: Regrid:
method: bilinear # Mandatory, str: Interpolation method. method: bilinear # Interpolation method (Mandatory, str)
type: to_system # Mandatory, str: 'to_system', 'to_reference', 'none', type: to_system # Interpolate to: 'to_system', 'to_reference', 'none',
# or CDO-accepted grid. # or CDO-accepted grid. (Mandatory, str)
Workflow: Workflow:
Calibration: Calibration:
method: mse_min # Mandatory, str: Calibration method. method: raw # Calibration method. (Mandatory, str)
save: 'none' # Options: 'all', 'none', 'exp_only', 'fcst_only' (Mandatory, str)
Anomalies: Anomalies:
compute: no # Mandatory, bool: Either yes/true or no/false compute: no # Either yes/true or no/false (Mandatory, bool)
cross_validation: no # Mandatory if 'compute: yes', bool: Either yes/true or no/false cross_validation: no # Either yes/true or no/false (Mandatory if 'compute: yes', bool)
save: 'fcst_only' # Options: 'all', 'none', 'exp_only', 'fcst_only' (Mandatory, str)
Downscaling:
# Assumption 1: leave-one-out cross-validation is always applied
# Assumption 2: for analogs, we select the best analog (minimum distance)
type: intbc # mandatory, 'none', 'int', 'intbc', 'intlr', 'analogs', 'logreg'.
int_method: conservative # regridding method accepted by CDO.
bc_method: bias # If type intbc. Options: 'bias', 'calibration', 'quantile_mapping', 'qm', 'evmos', 'mse_min', 'crps_min', 'rpc-based'.
lr_method: # If type intlr. Options: 'basic', 'large_scale', '4nn'
log_reg_method: # If type logreg. Options: 'ens_mean', 'ens_mean_sd', 'sorted_members'
target_grid: /esarchive/recon/ecmwf/era5/monthly_mean/tas_f1h/tas_200002.nc # nc file or grid accepted by CDO
nanalogs: # If type analgs. Number of analogs to be searched
save: 'all' # 'all'/'none'/'exp_only'
Indices:
## Indices available: NAO, Nino1+2, Nino3, Nino3.4, Nino4.
## Each index can only be computed if its area is within the selected region.
# obsproj: NAO computation method (see s2dv::NAO()) Default is yes/true. (Optional, bool)
# save: What to save. Options: 'all'/'none'. Default is 'all'.
# plot_ts: Generate time series plot? Default is yes/true. (Optional, bool)
# plot_sp: Generate spatial pattern plot? Default is yes/true. (Optional, bool)
NAO: {obsproj: yes, save: 'all', plot_ts: yes, plot_sp: yes}
Nino1+2: {save: 'all', plot_ts: yes, plot_sp: yes}
Nino3: {save: 'all', plot_ts: yes, plot_sp: yes}
Nino3.4: {save: 'all', plot_ts: yes, plot_sp: yes}
Nino4: {save: 'all', plot_ts: yes, plot_sp: yes}
Skill: Skill:
metric: RPSS CRPSS # Mandatory, str: List of skill metrics. metric: RPSS CRPSS # List of skill metrics separated by spaces or commas. (Mandatory, str)
save: 'all' # Options: 'all', 'none' (Mandatory, str)
Probabilities: Probabilities:
percentiles: [[1/3, 2/3], [1/10, 9/10], [1/4, 2/4, 3/4]] # Optional: Thresholds percentiles: [[1/3, 2/3], [1/10, 9/10], [1/4, 2/4, 3/4]] # Thresholds
# for quantiles and probability categories. # Each set of thresholds should be # for quantiles and probability categories. Each set of thresholds should be
# enclosed within brackets. # enclosed within brackets. (Optional)
save: 'percentiles_only' # Options: 'all', 'none', 'bins_only', 'percentiles_only' (Mandatory, str)
Visualization:
plots: skill_metrics, most_likely_terciles, forecast_ensemble_mean # Types of plots to generate (Optional, str)
multi_panel: yes # Multi-panel plot or single-panel plots. Default is 'no/false'. (Optional, bool)
projection: 'cylindrical_equidistant' # Options: 'cylindrical_equidistant', 'robinson', 'lambert_europe'. Default is cylindrical equidistant. (Optional, str)
mask_terciles: no # Whether to mask the non-significant points by rpss in the most likely tercile plot. yes/true, no/false or 'both'. Default is no/false. (Optional, str)
dots_terciles: yes # Whether to dot the non-significant by rpss in the most likely tercile plot. yes/true, no/false or 'both'. Default is no/false. (Optional, str)
Indicators: Indicators:
index: no # This feature is not implemented yet index: no # This feature is not implemented yet
ncores: 10 # Optional, int: number of cores to be used in parallel computation. ncores: 10 # Number of cores to be used in parallel computation.
# If left empty, defaults to 1. # If left empty, defaults to 1. (Optional, int)
remove_NAs: TRUE # Optional, bool: Whether to remove NAs. remove_NAs: yes # Whether to remove NAs.
# If left empty, defaults to FALSE. # If left empty, defaults to no/false. (Optional, bool)
Output_format: S2S4E # This feature is not implemented yet Output_format: 'S2S4E' # 'S2S4E' or 'Scorecards'. Determines the format of the output. Default is 'S2S4E'.
Run: Run:
Loglevel: INFO # Optional, str: 'DEBUG', 'INFO', 'WARN', 'ERROR' or 'FATAL'. Default 'INFO'. Loglevel: INFO # Minimum category of log messages to display: 'DEBUG', 'INFO', 'WARN', 'ERROR' or 'FATAL'.
Terminal: TRUE # Optional, bool: Whether to display log messages in the terminal. Default 'TRUE'. # Default value is 'INFO'. (Optional, str)
output_dir: /esarchive/scratch/vagudets/repos/sunset/out-logs/ # Mandatory, str: output directory Terminal: yes # Optional, bool: Whether to display log messages in the terminal.
code_dir: /esarchive/scratch/vagudets/repos/sunset/ # Default is yes/true.
output_dir: /esarchive/scratch/vagudets/repos/sunset-outputs/ # Output directory. Must have write permissions. (Mandatory, str)
code_dir: /esarchive/scratch/vagudets/repos/sunset/
``` ```
## List of /esarchive/ datasets ## List of /esarchive/ datasets
...@@ -87,28 +125,29 @@ Here is a list of the datasets that can currently be loaded by the tool. To requ ...@@ -87,28 +125,29 @@ Here is a list of the datasets that can currently be loaded by the tool. To requ
### Seasonal datasets ### Seasonal datasets
Systems: Systems:
| Forecast System | Monthly mean | Daily mean | /esarchive/exp/ path | | Forecast System | Monthly | Daily | /esarchive/exp/ path |
|----------------------------|-----------------|------------|-------------------------| |----------------------------|-----------------|------------|-------------------------|
| **ECMWF-SEAS5** | Yes | Yes | ecmwf/system5c3s/ | | **ECMWF-SEAS5** | Yes | Yes | ecmwf/system5c3s/ |
| **ECMWF-SEAS5.1** | Yes | Yes | ecmwf/system51c3s/ |
| **DWD-GCFS2.1** | Yes | No | dwd/system21_m1/ | | **DWD-GCFS2.1** | Yes | No | dwd/system21_m1/ |
| **CMCC-SPS3.5** | Yes | No | cmcc/system35c3s/ | | **CMCC-SPS3.5** | Yes | No | cmcc/system35c3s/ |
| **Meteo-France-System7** | Yes | No | meteofrance/system7c3s/ | | **Meteo-France-System7** | Yes | No | meteofrance/system7c3s/ |
| **JMA-CPS2** | Yes | No | jma/system2c3s/ | | **JMA-CPS2** | Yes | No | jma/system2c3s/ |
| **ECCC-CanCM4i** | May to November | No | eccc/eccc1/ | | **ECCC-CanCM4i** | May to November | No | eccc/eccc1/ |
| **UK-MetOffice-GloSea600** | Yes | No | ukmo/glosea6_system600-c3s/ | | **UK-MetOffice-GloSea600** | Yes | No | ukmo/glosea6_system600-c3s/ |
| **NCEP-CFSv2** | Yes | No | ncep/cfs-v2/ | | **NCEP-CFSv2** | Yes | No | ncep/cfs-v2/ |
Observations: Observations:
| Reference | Monthly mean | Daily mean | /esarchive/recon/ path | | Reference | Monthly | Daily | /esarchive/recon/ path |
|---------------|--------------|------------|------------------------| |---------------|--------------|------------|------------------------|
| **ERA5** | Yes | Yes | ecmwf/era5land/ | | **ERA5** | Yes | Yes | ecmwf/era5/ |
| **ERA5-Land** | `tas` only | Yes | ecmwf/era5land/ | | **ERA5-Land** | `tas` only | Yes | ecmwf/era5land/ |
| **UERRA** | No | `tas` only | ecmwf/uerra_mescan/ | | **UERRA** | No | `tas` only | ecmwf/uerra_mescan/ |
### Decadal datasets ### Decadal datasets
Systems: Systems:
| Forecast System | Monthly mean | Daily mean | forecast (DCPP-B)? | | Forecast System | Monthly | Daily | forecast (DCPP-B)? |
|------------------------|--------------|------------|--------------------| |------------------------|--------------|------------|--------------------|
| **BCC-CSM2-MR** | Yes | Yes | No | | **BCC-CSM2-MR** | Yes | Yes | No |
| **CanESM5** | Yes | Yes | Yes | | **CanESM5** | Yes | Yes | Yes |
...@@ -127,7 +166,7 @@ Systems: ...@@ -127,7 +166,7 @@ Systems:
| **NorCPM1-i2** | Yes | Yes | No | | **NorCPM1-i2** | Yes | Yes | No |
Observations: Observations:
| Reference | Monthly mean | Daily mean | | Reference | Monthly | Daily |
|---------------|--------------|------------| |---------------|--------------|------------|
| **GHCNv4** | Yes | No | | **GHCNv4** | Yes | No |
| **ERA5** | Yes | Yes | | **ERA5** | Yes | Yes |
...@@ -138,14 +177,14 @@ Observations: ...@@ -138,14 +177,14 @@ Observations:
## The first steps ## The first steps
In order to run the ESS Verification Suite, you need to load the necessary modules. To do this, you can run the command `source MODULES` in the terminal, from the main folder of the SUNSET repository. In order to run the SUNSET Suite, you need to load the necessary modules (R, CDO, GDAL, GEOS and PROJ). To do this, you can run the command `source MODULES` in the terminal, from the main folder of the SUNSET repository.
Before calling the modules in your script or in the R console, you should run the `prepare_outputs()` function as shown in the [example script](https://earth.bsc.es/gitlab/es/sunset/-/snippets/96), which will read your recipe and set up the directory for your outputs. Before calling the modules in your script or in the R console, you should run the `prepare_outputs()` function as shown in the [example script](https://earth.bsc.es/gitlab/es/sunset/-/snippets/96), which will read your recipe and set up the directory for your outputs.
`prepare_outputs()` will perform a check on your recipe to detect potential errors. If you want to disable this check, you may set the argument `disable_checks = TRUE` when calling the function. `prepare_outputs()` will perform a check on your recipe to detect potential errors. If you want to disable this check, you may set the argument `disable_checks = TRUE` when calling the function.
If you had a recipe named recipe-wiki.yml that looked like the example in this wiki, this directory might look something like this: If you had a recipe named 'recipe-wiki.yml' that looked like the example in this wiki, this directory might look something like this:
`/esarchive/scratch/vagudets/repos/sunset/out-logs/recipe-wiki_20221025164151` `/esarchive/scratch/vagudets/repos/sunset-outputs/recipe-wiki_20221025164151/`
Inside you will find a log file, a copy of your recipe, and your outputs from the Saving and Visualization modules. Inside you will find a log file, a copy of your recipe, and your outputs from the Saving and Visualization modules.
...@@ -153,13 +192,13 @@ Inside you will find a log file, a copy of your recipe, and your outputs from th ...@@ -153,13 +192,13 @@ Inside you will find a log file, a copy of your recipe, and your outputs from th
The Loading module retrieves the data requested in the recipe from /esarchive/, interpolates it to the desired grids if interpolation has been requested, and converts it to objects of class `s2dv_cube`, which can be passed onto the other modules in the tool. An `s2dv_cube` object is a list containing the data array in the element $data and many other elements that store the metadata. The Loading module retrieves the data requested in the recipe from /esarchive/, interpolates it to the desired grids if interpolation has been requested, and converts it to objects of class `s2dv_cube`, which can be passed onto the other modules in the tool. An `s2dv_cube` object is a list containing the data array in the element $data and many other elements that store the metadata.
The output of the main function, `load_datasets()`, is a list containing the hindcast, observations and forecast, named hcst, obs and fcst respectively. fcst will be `NULL` if no forecast years have been requested. The output of the main function, `Loading()`, is a list containing the hindcast, observations and forecast, named hcst, obs and fcst respectively. fcst will be `NULL` if no forecast years have been requested.
### Regridding ### Regridding
The Loading module can interpolate the data while loading them, and the core tool is CDO. The interpolation methods that can be specified in the recipe under Regrid:method are those accepted by CDO: "conservative", "bilinear", "bicubic", "distance-weighted", "con2", "laf", "nn". See CDO User Guide for more details. The Loading module can interpolate the data while loading them, using CDO as the core tool. The interpolation methods that can be specified in the recipe under `Regrid:method` are those accepted by CDO: `'conservative'`, `'bilinear'`,`'bicubic'`, `'distance-weighted'`, `'con2'`, `'laf'` and `'nn'`. Consult the CDO User Guide for more details.
The target grid is to be specified in Regrid:type. The options are: The target grid is to be specified in `Regrid:type`. The options are:
- `‘to_system’`: The observations are interpolated to the system grid. - `‘to_system’`: The observations are interpolated to the system grid.
- `‘to_reference’:` the hindcast and forecast are interpolated to the reference grid. - `‘to_reference’:` the hindcast and forecast are interpolated to the reference grid.
- `‘none’`: No interpolation is done when loading the data. - `‘none’`: No interpolation is done when loading the data.
...@@ -167,39 +206,64 @@ The target grid is to be specified in Regrid:type. The options are: ...@@ -167,39 +206,64 @@ The target grid is to be specified in Regrid:type. The options are:
### Loading the desired latitude and longitude range ### Loading the desired latitude and longitude range
The Verification Suite uses startR to load the data, which you may need to take into account to make sure you can load your desired region or range. For example, setting `lonmin: 0` and `lonmax: 360` can be problematic because Start() will consider them the same point. The Verification Suite uses [startR](https://CRAN.R-project.org/package=startR) to load the data, which you may need to take into account to make sure you can load your desired region or range. For example, setting `lonmin: 0` and `lonmax: 360` can be problematic because the `Start()` function will consider them the same point.
The startR documentation has [a guide to explain how to select your longitude range appropriately](https://earth.bsc.es/gitlab/es/startR/-/blob/master/inst/doc/faq.md#11-select-the-longitudelatitude-region). The Loading module handles the `Sort()` and `CircularSort()` automatically from the ranges in your recipe.
The StartR documentation has [a guide to explain how to select your longitude range appropriately](https://earth.bsc.es/gitlab/es/startR/-/blob/master/inst/doc/faq.md#11-select-the-longitudelatitude-region). The Loading module handles the `Sort()` and `CircularSort()` automatically from the ranges in your recipe. ## Units module
The Units module should be used by calling the Relocation leaveUnits() function after the Loading module to ensure consistency between system and reference datasets. It will verify that the hindcast, forecast and observations share the same units, and make the appropriate transformations if there are any differences.
Additionally, user-specified units can be included in the recipe for each variable. Currently, the units of three variables can be converted. The following units are considered:
- **Temperature:** `"C", "K"`
- **Precipitation:** `"ms-1", "kgm-2s-1", "mm", "m"` For the precipitation units, the user can also decide whether to obtain the total amount of the flux by setting parameter`flux` to true/false in the Variable section of the recipe.
- **Pressure:** `"pa", "hpa", "mb"`
## Calibration module ## Calibration module
The Calibration module performs bias correction on the loaded data. It accepts the output of the Loading module as input, and also requires the recipe. It applies a calibration method to the hindcast and forecast data using the observations as a reference, and returns the calibrated data and its metadata as an `s2dv_cube` object. The Calibration module performs bias correction on the loaded data. It accepts the output of the Loading module as input, and also requires the recipe. It applies the selected bias correction method to the hindcast and forecast data using the observations as a reference, and returns the calibrated data and its metadata as an `s2dv_cube` object.
The output of the main function, `calibrate_datasets()`, is a list containing the calibrated hindcast and forecast, named hcst and fcst respectively. fcst will be `NULL` if no forecast years have been requested. The output of the main function, `Calibration()`, is a list containing the calibrated hindcast and forecast, named hcst and fcst respectively. fcst will be `NULL` if no forecast years have been requested.
### Calibration methods currently available: ### Calibration methods currently available:
The calibration method can be requested in the Workflow:Calibration:method section of the recipe. **The user can only request one calibration method per recipe.** This is a list of the methods currently available: The calibration method can be requested in the `Workflow:Calibration:method` section of the recipe. **The user can only request one calibration method per recipe.** This is a list of the methods currently available:
- `'raw'`: No calibration is performed. A warning will show up on the terminal when `calibrate_datasets()` is called, and it will return the uncalibrated hindcast and forecast. - `'raw'`: No calibration is performed. A warning will show up on the terminal when `Calibration()` is called, and it will return the uncalibrated hindcast and forecast.
- Daily data: Quantile mapping `'qmap'`. - Daily data: Quantile mapping `'qmap'`.
For more details, see the [CSTools documentation](https://CRAN.R-project.org/package=CSTools) for CST_QuantileMapping(). For more details, see the [CSTools documentation](https://CRAN.R-project.org/package=CSTools) for `CST_QuantileMapping()`.
- Monthly data: `'bias'`, `'evmos'`, `'mse_min'`, `'crps_min'`, and `'rpc-based'`. - Monthly data: `'bias'`, `'evmos'`, `'mse_min'`, `'crps_min'`, and `'rpc-based'`.
For more details, see the [CSTools documentation](https://CRAN.R-project.org/package=CSTools) for CST_Calibration(). For more details, see the [CSTools documentation](https://CRAN.R-project.org/package=CSTools) for `CST_Calibration()`.
## Anomalies module ## Anomalies module
The Anomalies module computes the anomalies of the data with respect to the climatological mean; with or without cross-validation, depending on what it specified in the recipe. Calibration performs bias correction on the loaded data. It accepts the output of either the Loading or the Calibration module as input, and also requires the recipe. It makes use of the CSTools function CST_Calibration(). The Anomalies module computes the anomalies of the data with respect to the climatological mean; with or without cross-validation, depending on what it specified in the recipe. It accepts the output of either the Loading or the Calibration module as input, and also requires the recipe. It makes use of the CSTools function `CST_Anomaly()`.
The output of the main function, `Anomalies()`, is a list of `s2dv_cube` objects containing the anomalies for the hcst, fcst and obs, as well as the original hcst and obs full fields in case they are needed for later computations.
## Indices module
The Indices module aggregates the hindcast and reference data to compute climatological indices such as the North Atlantic Oscillation (NAO) or El Niño indices.
The main function, `Indices()`, returns the hcst and obs s2dv_cube objects for each requested index, in the form of a list of lists. The 'latitude' and 'longitude' dimensions of the original arrays are aggregated into a single 'region' dimension.
Indices currently available:
| Index | Recipe name | Longitude range | Latitude range |
|--------------------------------|---------------|-------------------|------------------|
| **North Atlantic Oscillation** | NAO | -80º to 40º | 20º to 80º |
| **Niño 1+2** | Nino1+2 | -90º to -80º | -10º to 0º |
| **Niño 3** | Nino3 | -150º to -90º | -5º to 5º |
| **Niño 3.4** | Nino3.4 | -170º to -120º | -5º to 5º |
| **Niño 4** | Nino4 | 160º to -150º | -5º to 5º |
The output of the main function, `compute_anomalies()`, is a list of `s2dv_cube` objects containing the anomalies for the hcst, fcst and obs, as well as the original hcst and obs full fields in case they are needed for later computations.
## Skill module ## Skill module
The Skill module is the part of the workflow that computes the metrics to assess the quality of a forecast. It accepts the output of the Calibration module as input, and also requires the recipe. It is comprised of two main functions: The Skill module is the part of the workflow that computes the metrics to assess the quality of a forecast. It accepts the output of the Calibration module as input, and also requires the recipe. It is comprised of two main functions:
**compute_skill_metrics()**: Computes the verification metrics requested in Workflow:Skill:metric. The user can request an unlimited number of verification metrics per recipe. The following metrics are currently available: **Skill()**: Computes the verification metrics requested in `Workflow:Skill:metric`. The user can request an unlimited number of verification metrics per recipe. The following metrics are currently available:
- `EnsCorr`: Ensemble Mean Correlation. - `EnsCorr`: Ensemble Mean Correlation.
- `Corr`: Ensemble Correlation. - `Corr`: Ensemble Correlation.
...@@ -214,12 +278,14 @@ The Skill module is the part of the workflow that computes the metrics to assess ...@@ -214,12 +278,14 @@ The Skill module is the part of the workflow that computes the metrics to assess
- `Mean_Bias`: Mean bias of the ensemble. - `Mean_Bias`: Mean bias of the ensemble.
- `Mean_Bias_SS`: Mean Bias Skill Score. - `Mean_Bias_SS`: Mean Bias Skill Score.
- `EnsSprErr`: Ensemble Spread-to-Error Ratio. - `EnsSprErr`: Ensemble Spread-to-Error Ratio.
- `MSSS`: Mean Square Skill Score
- `MSE`: Mean Square Error
**Note**: For the following metrics: "EnsCorr", "FRPS", "RPSS", "FRPSS", "BSS90", "BSS10", if “_specs” is added at the end of the metric (e.g. RPSS_specs), it will be computed using SpecsVerification. **Note**: For the following metrics: "EnsCorr", "FRPS", "RPSS", "FRPSS", "BSS90", "BSS10", if “_specs” is added at the end of the metric (e.g. RPSS_specs), the values will be computed using SpecsVerification.
The output of compute_skill_metrics() is a list containing one or more arrays with named dimensions; usually ‘time’, ‘longitude’ and ‘latitude’. For more details on the specific output for each metric, see the documentation for [s2dv](https://CRAN.R-project.org/package=s2dv) and [SpecsVerification](https://CRAN.R-project.org/package=SpecsVerification). The output of `Skill()` is a list containing one or more arrays with named dimensions; usually 'var', ‘time’, ‘longitude’ and ‘latitude’. For more details on the specific output for each metric, see the documentation for [s2dv](https://CRAN.R-project.org/package=s2dv) and [SpecsVerification](https://CRAN.R-project.org/package=SpecsVerification).
**compute_probabilities()** returns a list of two elements with arrays containing the values corresponding to the thresholds in Workflow:Probabilities:percentiles (\$percentiles), as well as their probability bins (\$probs). Each list contains arrays with named dimensions ‘time’, ‘longitude’ and ‘latitude’. **Probabilities()** returns a list of two elements with arrays containing the values corresponding to the thresholds in `Workflow:Probabilities:percentiles` (`$percentiles`), as well as their probability bins (`$probs`). Each list contains arrays with named dimensions ‘time’, ‘longitude’ and ‘latitude’.
For example, if the extremes (1/10, 9/10) are requested, the output will be: For example, if the extremes (1/10, 9/10) are requested, the output will be:
`$percentiles`: `$percentiles`:
...@@ -238,47 +304,421 @@ For example, if the extremes (1/10, 9/10) are requested, the output will be: ...@@ -238,47 +304,421 @@ For example, if the extremes (1/10, 9/10) are requested, the output will be:
**Note**: When naming the variables, the probability thresholds are converted to percentiles and rounded to the nearest integer to avoid dots in variable or file names. However, this is just a naming convention; the computations are performed based on the original thresholds specified in the recipe. **Note**: When naming the variables, the probability thresholds are converted to percentiles and rounded to the nearest integer to avoid dots in variable or file names. However, this is just a naming convention; the computations are performed based on the original thresholds specified in the recipe.
## Saving module ## Saving
The Saving module contains several functions that export the data (the calibrated hindcast and forecast, the corresponding observations, the skill metrics, percentiles and probabilities) to netCDF files and save them. The Saving module contains several functions that export the data (the calibrated hindcast and forecast, the corresponding observations, the skill metrics, percentiles and probabilities) to netCDF files and save them.
`save_data()` serves as the main wrapper function for this module.`recipe` (the recipe) and `data` (the list of `s2dv_cube` objects containing at least the processed hindcast and observation data). The rest of the arguments (skill_metrics and probabilities) are optional. Including `skill_metrics` and `probabilities` will save the skill metrics and the percentiles and probability bins, respectively. Since version 1.x.x of SUNSET, each module has different options to save all or some the data produced. These options can be specified in the recipe. To see the options for a specific module, consult the 'Recipe' section of this wiki or the documentation for that module. Once the computations are finished, the output subdirectories and files are generated internally within the module function, according to the user's specifications.
Inside the main output directory, the netCDF files produced by each module will be stored inside `/outputs/<module_name>/`. The old function `save_data()`, which has been renamed to `Saving()`, still exists, but will likely be deprecated in the near future.
### The structure of the output directory
The outputs are saved to a unique folder inside the directory you specified in the recipe, under `outputs/`. For example, if you are saving the metrics computed with the Skill module, they will be stored inside `<output_dir>/outputs/Skill/`, where <output_dir> is the output directory specified in the recipe.
The structure of the subdirectories and file names will depend on the selected '`Output_format`' in the recipe.
There are two possible output formats currently: `'S2S4E'` and `'Scorecards'`, described below. To request the inclusion of an additional output format, please open an issue. It is also possible to save the data to a custom directory in a custom format, by making use of the CSTools functions `SaveExp()` and `CST_SaveExp()`.
#### S2S4E (default)
With the 'S2S4E' output format, the structure of the subdirectory and the files is as follows:
If fcst_year has been requested:
- `<system>/<calibration_method>-<frequency>/<forecast_date>/<var>/`
If fcst_year is empty:
- For seasonal data: `<system>/<calibration_method>-<frequency>/hcst-<mmdd>/<var>/`
- For decadal data: `<system>/<calibration_method>-<frequency>/hcst-<yyyy>_<yyyy>/<var>/`
For example, in our example recipe, the calibrated tas netCDF Files will be in:
`/esarchive/scratch/vagudets/repos/sunset-outputs/recipe-wiki_20221025164151/outputs/Calibration/mse_min-monthly_mean/20201101/tas/`
The structure of the names of the files is:
- Skill metrics: `<var>-skill_month<mm>.nc`
- Correlation: `<var>-corr_month<mm>.nc`
- Processed hindcast and forecast data: `<var>_<yyyymmdd>.nc`
- Observations: `<var>-obs_<yyyymmdd>.nc`
- Probabilities: `<var>-probs_<yyyymmdd>.nc`
- Percentiles: `<var>-percentiles_month<mm>.nc`
Where `var` is the name of the variable, `yyyy` is the year, `mm` is the initialization month and `mm` is the initialization day.
#### Scorecards:
For the 'Scorecards' output format, the structure of the subdirectory and the files is:
- Skill metrics: `<system>/<variable>/scorecards_<system>-<reference>_<variable>-skill_<startyear>_<endyear>_s<mm>.nc`
- Processed hindcast and forecast data: `<system>/<variable>/scorecards_<system>-<reference>_<variable>_<yyyymmdd>_<startyear>_<endyear>_s<mm>.nc`
- Observations: `<system>/<variable>/scorecards_<system>-<reference>_<variable>-obs_<yyyymmdd>_<startyear>_<endyear>_s<mm>.nc`
- Probabilities: `<system>/<variable>/scorecards_<system>-<reference>_<variable>-probs_<yyyymmdd>_<startyear>_<endyear>_s<mm>.nc`
- Percentiles: `<system>/<variable>/scorecards_<system>-<reference>_<variable>-percentiles_<startyear>_<endyear>_s<mm>.nc`
`system` and `reference`are abbreviations of the names of the system and reference, respectively.
`var` is the name of the variable, `startyear` and `endyear` are the initial and final hindcast years, `yyyymmdd` is the start date for each year and `mm` is the initialization month.
### The Saving() wrapper function (to be deprecated)
`Saving()` serves as the main wrapper function for this module.`recipe` (the recipe) and `data` (the list of `s2dv_cube` objects containing at least the processed hindcast and observation data). The rest of the arguments (skill_metrics and probabilities) are optional. Including `skill_metrics` and `probabilities` will save the skill metrics and the percentiles and probability bins, respectively.
```R ```R
# Therefore, all the data can be saved at once: # Therefore, all the data can be saved at once:
save_data(recipe, data, skill_metrics, probabilities) Saving(recipe, data, skill_metrics, probabilities)
# Or, one can choose to save only some of it. For example, # Or, one can choose to save only some of it. For example,
# saving the calibrated data and corresponding observations only: # saving the calibrated data and corresponding observations only:
save_data(recipe, data) Saving(recipe, data)
# Or saving the hcst/fcst/obs and skill metrics only: # Or saving the hcst/fcst/obs and skill metrics only:
save_data(recipe, data, skill_metrics = skill_metrics) Saving(recipe, data, skill_metrics = skill_metrics)
# Or only hcst/fcst/obs and the probabilities: # Or only hcst/fcst/obs and the probabilities:
save_data(recipe, data, probabilities = probabilities) Saving(recipe, data, probabilities = probabilities)
```
## Visualization module
The Visualization module provides a few basic plots to visualize the data loaded and computed using the previous modules. The color palettes and titles of the plots will vary depending on the output format requested in the recipe. The palettes are generated using the clim.colors() function from the [s2dv](https://cran.r-project.org/web/packages/s2dv/index.html) package.
- For 'S2S4E', the "bluered" palette is used.
- For 'Scorecards', "purpleorange" is used.
**Visualization()** is the main wrapper function for this module. It generates the plots and saves them to the same output directory described in the Saving module section, under `plots/`.
This function's parameters are similar to those in the Saving module:
`recipe` (the recipe) and `data` (the list obtained from the Loading module) are mandatory arguments. The rest of the arguments (`skill_metrics`, `probabilities`, and `significance`) are optional:
- `skill_metrics`: List in the format of the Skill module output, containing the skill metrics as named arrays.
- `probabilities`: List in the format of the Skill module output, containing the 33rd and 66th percentiles.
- If `significance = TRUE` (default), the statistical significance dots will be displayed in the plot, when available. It defaults to FALSE.
`Visualization()` attempts to generate:
- Plots of all the skill metrics provided in `skill_metrics`.
- A plot of the forecast ensemble mean, if a forecast has been provided.
- A Most Likely Terciles plot, if the `probabilities` include the terciles (percentiles 33 and 66).
The three functions that `Visualization()` calls can also be called independently. In this case, archive configuration file and the the output directory for the plots will have to be provided explicitly as arguments to :
**plot_skill_metrics(recipe, archive, data_cube, skill_metrics, outdir, significance = F)**: Generates, for each metric in skill_metrics, a figure with one plot per time step, and saves each figure to the output directory `outdir` as `<metric>.png`.
`data_cube` is an s2dv_cube containing the appropriate metadata, for example the hcst object that was passed to the Skill module.
**plot_ensemble_mean(recipe, archive, fcst, outdir)**: Computes the forecast ensemble mean and generates a figure with one plot per time step, and saves it to the output directory `outdir` as `forecast_ensemble_mean.png`
**plot_most_likely_terciles(recipe, archive, fcst, percentiles, outdir)**: Computes the forecast tercile probability bins with respect to the terciles provided in 'percentiles', then generates a figure with one plot per time step and saves it to the directory `outdir` as `forecast_most_likely_terciles.png`
## Introduction
This the GitLab repository for the SUNSET (SUbseasoNal to decadal climate forecast post-processIng and asSEmenT suite), a modular tool for subseasonal to seasonal to decadal forecast verification workflows. It is intended to have a modularized structure, where each module is a separate part of the code that performs a specific task, so that parts of the workflow can be skipped or reordered.
The datasets, forecast horizon, time period, skill metrics to compute and other parameters are specified by the user in a configuration file, called "recipe".
After definining the recipe, the user can create a script using the functions available in the tool, which use apply the information in the recipe to do the desired computations. You an example script [in the SUNSET code Snippets](https://earth.bsc.es/gitlab/es/sunset/-/snippets/111).
- Modules currently available: Loading, Calibration, Anomalies, Skill, Indices, Saving, Visualization
- Modules in development: Downscaling, Scorecards
- Future modules: Aggregation, Indicators
This tool is in the early stages of development, so the code and the information in this wiki may be subject to frequent changes and updates. This wiki contains all the information needed to use the available modules.
## Recipes
In order to use SUNSET, users must define a recipe containing all the information pertaining to their workflow.
Here is an example of a recipe to load monthly mean ECMWF System 5 data from `/esarchive/`, with a 1993 to 2016 hindcast period, the corresponding ERA5 observations, and a 2020 forecast for the November initialization, for the months of November and December.
Following this recipe, the observations will be interpolated to the experiment grid (Regrid type: 'to_system') using bilinear interpolation. The hindcast and forecast will be calibrated using Quantile Mapping, and the Ranked Probability Skill Score (RPSS) and Continuous Ranked Probability Skill Score (CRPSS) will be computed.
The terciles (1/3, 2/3), quartiles (1/4, 2/4, 3/4), extremes (1/10, 9/10) and their corresponding probability bins will also be computed. Any output files will be saved inside the output directory.
```yaml
Description:
Author: V. Agudetse
Info: ECMWF System5 Seasonal Forecast Example recipe (monthly mean, tas)
Analysis:
Horizon: seasonal # Mandatory, str: 'subseasonal', 'seasonal', or 'decadal'
Variables:
# name: variable name(s) in the /esarchive (Mandatory, str)
# freq: 'monthly_mean', 'daily' or 'daily_mean' (Mandatory, str)
- {name: 'tas', freq: 'monthly_mean'}
Datasets:
System:
# name: System name (Mandatory, str)
# member: 'all' or individual members, separated by a comma and in quotes (decadal only, str)
- {name: 'ECMWF-SEAS5', member: 'all'}
Multimodel: no # Either yes/true or no/false (Mandatory, bool)
Reference:
- {name: 'ERA5'} # Reference name (Mandatory, str)
Time:
sdate: '1101' # Start date, 'mmdd' (Mandatory, int)
fcst_year: '2020' # Forecast initialization year 'YYYY' (Optional, int)
hcst_start: '1993' # Hindcast initialization start year 'YYYY' (Mandatory, int)
hcst_end: '2016' # Hindcast initialization end year 'YYYY' (Mandatory, int)
ftime_min: 1 # First forecast time step in months. Starts at “1”. (Mandatory, int)
ftime_max: 6 # Last forecast time step in months. Starts at “1”. (Mandatory, int)
Region:
latmin: -90 # minimum latitude (Mandatory, int)
latmax: 90 # maximum latitude (Mandatory, int)
lonmin: 0 # minimum longitude (Mandatory, int)
lonmax: 359.9 # maximum longitude (Mandatory, int)
Regrid:
method: bilinear # Interpolation method (Mandatory, str)
type: to_system # Interpolate to: 'to_system', 'to_reference', 'none',
# or CDO-accepted grid. (Mandatory, str)
Workflow:
Calibration:
method: mse_min # Calibration method. (Mandatory, str)
save: 'all' # Options: 'all', 'none', 'exp_only', 'fcst_only' (Mandatory, str)
Anomalies:
compute: no # Either yes/true or no/false (Mandatory, bool)
cross_validation: no # Either yes/true or no/false (Mandatory if 'compute: yes', bool)
save: 'fcst_only' # Options: 'all', 'none', 'exp_only', 'fcst_only' (Mandatory, str)
Skill:
metric: RPSS CRPSS # List of skill metrics separated by spaces or commas. (Mandatory, str)
save: 'all' # Options: 'all', 'none' (Mandatory, str)
Probabilities:
percentiles: [[1/3, 2/3], [1/10, 9/10], [1/4, 2/4, 3/4]] # Thresholds
# for quantiles and probability categories. Each set of thresholds should be
# enclosed within brackets. (Optional)
save: 'percentiles_only' # Options: 'all', 'none', 'bins_only', 'percentiles_only' (Mandatory, str)
Indices:
## Indices available: NAO, Nino1+2, Nino3, Nino3.4, Nino4.
## Each index can only be computed if its area is within the selected region.
# obsproj: NAO computation method (see s2dv::NAO()) Default is yes/true. (Optional, bool)
# save: What to save. Options: 'all'/'none'. Default is 'all'.
# plot_ts: Generate time series plot? Default is yes/true. (Optional, bool)
# plot_sp: Generate spatial pattern plot? Default is yes/true. (Optional, bool)
NAO: {obsproj: yes, save: 'all', plot_ts: yes, plot_sp: yes}
Nino1+2: {save: 'all', plot_ts: yes, plot_sp: yes}
Nino3: {save: 'all', plot_ts: yes, plot_sp: yes}
Nino3.4: {save: 'all', plot_ts: yes, plot_sp: yes}
Nino4: {save: 'all', plot_ts: yes, plot_sp: yes}
Visualization:
plots: skill_metrics, most_likely_terciles, forecast_ensemble_mean # Types of plots to generate (Optional, str)
multi_panel: yes # Multi-panel plot or single-panel plots. Default is 'no/false'. (Optional, bool)
projection: 'cylindrical_equidistant' # Options: 'cylindrical_equidistant', 'robinson', 'lambert_europe'. Default is cylindrical equidistant. (Optional, str)
mask_terciles: no # Whether to mask the non-significant points by rpss in the most likely tercile plot. yes/true, no/false or 'both'. Default is no/false. (Optional, str)
dots_terciles: yes # Whether to dot the non-significant by rpss in the most likely tercile plot. yes/true, no/false or 'both'. Default is no/false. (Optional, str)
Indicators:
index: no # This feature is not implemented yet
ncores: 10 # Number of cores to be used in parallel computation.
# If left empty, defaults to 1. (Optional, int)
remove_NAs: yes # Whether to remove NAs.
# If left empty, defaults to no/false. (Optional, bool)
Output_format: 'S2S4E' # 'S2S4E' or 'Scorecards'. Determines the format of the output. Default is 'S2S4E'.
Run:
Loglevel: INFO # Minimum category of log messages to display: 'DEBUG', 'INFO', 'WARN', 'ERROR' or 'FATAL'.
# Default value is 'INFO'. (Optional, str)
Terminal: yes # Optional, bool: Whether to display log messages in the terminal.
# Default is yes/true.
output_dir: /esarchive/scratch/vagudets/repos/sunset-outputs/ # Output directory. Must have write permissions. (Mandatory, str)
code_dir: /esarchive/scratch/vagudets/repos/sunset/
``` ```
More flexibility will be added in future releases.
## List of /esarchive/ datasets
Here is a list of the datasets that can currently be loaded by the tool. To request that an additional dataset be added, please [open an issue](https://earth.bsc.es/gitlab/es/sunset/-/issues).
### Seasonal datasets
Systems:
| Forecast System | Monthly mean | Daily mean | /esarchive/exp/ path |
|----------------------------|-----------------|------------|-------------------------|
| **ECMWF-SEAS5** | Yes | Yes | ecmwf/system5c3s/ |
| **ECMWF-SEAS5.1** | Yes | Yes | ecmwf/system51c3s/ |
| **DWD-GCFS2.1** | Yes | No | dwd/system21_m1/ |
| **CMCC-SPS3.5** | Yes | No | cmcc/system35c3s/ |
| **Meteo-France-System7** | Yes | No | meteofrance/system7c3s/ |
| **JMA-CPS2** | Yes | No | jma/system2c3s/ |
| **ECCC-CanCM4i** | May to November | No | eccc/eccc1/ |
| **UK-MetOffice-GloSea600** | Yes | No | ukmo/glosea6_system600-c3s/ |
| **NCEP-CFSv2** | Yes | No | ncep/cfs-v2/ |
Observations:
| Reference | Monthly mean | Daily mean | /esarchive/recon/ path |
|---------------|--------------|------------|------------------------|
| **ERA5** | Yes | Yes | ecmwf/era5/ |
| **ERA5-Land** | `tas` only | Yes | ecmwf/era5land/ |
| **UERRA** | No | `tas` only | ecmwf/uerra_mescan/ |
### Decadal datasets
Systems:
| Forecast System | Monthly mean | Daily mean | forecast (DCPP-B)? |
|------------------------|--------------|------------|--------------------|
| **BCC-CSM2-MR** | Yes | Yes | No |
| **CanESM5** | Yes | Yes | Yes |
| **CESM1-1-CAM5-CMIP5** | Yes | WIP | No |
| **CMCC-CM2-SR5** | Yes | Yes | Yes |
| **EC-Earth3-i1** | Yes | Yes | No |
| **EC-Earth3-i2** | Yes | Yes | No |
| **EC-Earth3-i4** | Yes | Yes | Yes |
| **HadGEM3-GC3.1-MM** | Yes | Yes | Yes |
| **IPSL-CM6A-LR** | Yes | Yes | No |
| **MIROC6** | Yes | Yes | No |
| **MPI-ESM1.2-HR** | Yes | Yes | No |
| **MPI-ESM1.2-LR** | Yes | No | Yes |
| **MRI-ESM2-0** | Yes | Yes | No |
| **NorCPM1-i1** | Yes | Yes | No |
| **NorCPM1-i2** | Yes | Yes | No |
Observations:
| Reference | Monthly mean | Daily mean |
|---------------|--------------|------------|
| **GHCNv4** | Yes | No |
| **ERA5** | Yes | Yes |
| **JRA-55** | Yes | Yes |
| **GISTEMPv4** | Yes | No |
| **HadCRUT4** | Yes | No |
| **HadSLP2** | Yes | No |
## The first steps
In order to run the SUNSET Suite, you need to load the necessary modules (R, CDO, GDAL, GEOS and PROJ). To do this, you can run the command `source MODULES` in the terminal, from the main folder of the SUNSET repository.
Before calling the modules in your script or in the R console, you should run the `prepare_outputs()` function as shown in the [example script](https://earth.bsc.es/gitlab/es/sunset/-/snippets/96), which will read your recipe and set up the directory for your outputs.
`prepare_outputs()` will perform a check on your recipe to detect potential errors. If you want to disable this check, you may set the argument `disable_checks = TRUE` when calling the function.
If you had a recipe named 'recipe-wiki.yml' that looked like the example in this wiki, this directory might look something like this:
`/esarchive/scratch/vagudets/repos/sunset-outputs/recipe-wiki_20221025164151/`
Inside you will find a log file, a copy of your recipe, and your outputs from the Saving and Visualization modules.
## Loading module
The Loading module retrieves the data requested in the recipe from /esarchive/, interpolates it to the desired grids if interpolation has been requested, and converts it to objects of class `s2dv_cube`, which can be passed onto the other modules in the tool. An `s2dv_cube` object is a list containing the data array in the element $data and many other elements that store the metadata.
The output of the main function, `Loading()`, is a list containing the hindcast, observations and forecast, named hcst, obs and fcst respectively. fcst will be `NULL` if no forecast years have been requested.
### Regridding
The Loading module can interpolate the data while loading them, using CDO as the core tool. The interpolation methods that can be specified in the recipe under `Regrid:method` are those accepted by CDO: `'conservative'`, `'bilinear'`,`'bicubic'`, `'distance-weighted'`, `'con2'`, `'laf'` and `'nn'`. Consult the CDO User Guide for more details.
The target grid is to be specified in `Regrid:type`. The options are:
- `‘to_system’`: The observations are interpolated to the system grid.
- `‘to_reference’:` the hindcast and forecast are interpolated to the reference grid.
- `‘none’`: No interpolation is done when loading the data.
- A CDO-accepted grid format, such as `'r360x180'` or the link to a netCDF file. In this case, both the system and the reference will be interpolated to this grid. See the FAQs section for more details.
### Loading the desired latitude and longitude range
The Verification Suite uses [startR](https://CRAN.R-project.org/package=startR) to load the data, which you may need to take into account to make sure you can load your desired region or range. For example, setting `lonmin: 0` and `lonmax: 360` can be problematic because the `Start()` function will consider them the same point.
The startR documentation has [a guide to explain how to select your longitude range appropriately](https://earth.bsc.es/gitlab/es/startR/-/blob/master/inst/doc/faq.md#11-select-the-longitudelatitude-region). The Loading module handles the `Sort()` and `CircularSort()` automatically from the ranges in your recipe.
## Calibration module
The Calibration module performs bias correction on the loaded data. It accepts the output of the Loading module as input, and also requires the recipe. It applies the selected bias correction method to the hindcast and forecast data using the observations as a reference, and returns the calibrated data and its metadata as an `s2dv_cube` object.
The output of the main function, `Calibration()`, is a list containing the calibrated hindcast and forecast, named hcst and fcst respectively. fcst will be `NULL` if no forecast years have been requested.
### Calibration methods currently available:
The calibration method can be requested in the `Workflow:Calibration:method` section of the recipe. **The user can only request one calibration method per recipe.** This is a list of the methods currently available:
- `'raw'`: No calibration is performed. A warning will show up on the terminal when `Calibration()` is called, and it will return the uncalibrated hindcast and forecast.
- Daily data: Quantile mapping `'qmap'`.
For more details, see the [CSTools documentation](https://CRAN.R-project.org/package=CSTools) for `CST_QuantileMapping()`.
- Monthly data: `'bias'`, `'evmos'`, `'mse_min'`, `'crps_min'`, and `'rpc-based'`.
For more details, see the [CSTools documentation](https://CRAN.R-project.org/package=CSTools) for `CST_Calibration()`.
## Anomalies module
The Anomalies module computes the anomalies of the data with respect to the climatological mean; with or without cross-validation, depending on what it specified in the recipe. It accepts the output of either the Loading or the Calibration module as input, and also requires the recipe. It makes use of the CSTools function `CST_Anomaly()`.
The output of the main function, `Anomalies()`, is a list of `s2dv_cube` objects containing the anomalies for the hcst, fcst and obs, as well as the original hcst and obs full fields in case they are needed for later computations.
## Indices module
The Indices module aggregates the hindcast and reference data to compute climatological indices such as the North Atlantic Oscillation (NAO) or El Niño indices.
The main function, `Indices()`, returns the hcst and obs s2dv_cube objects for each requested index, in the form of a list of lists. The 'latitude' and 'longitude' dimensions of the original arrays are aggregated into a single 'region' dimension.
Indices currently available:
| Index | Recipe name | Longitude range | Latitude range |
|--------------------------------|---------------|-------------------|------------------|
| **North Atlantic Oscillation** | NAO | -80º to 40º | 20º to 80º |
| **Niño 1+2** | Nino1+2 | -90º to -80º | -10º to 0º |
| **Niño 3** | Nino3 | -150º to -90º | -5º to 5º |
| **Niño 3.4** | Nino3.4 | -170º to -120º | -5º to 5º |
| **Niño 4** | Nino4 | 160º to -150º | -5º to 5º |
## Skill module
The Skill module is the part of the workflow that computes the metrics to assess the quality of a forecast. It accepts the output of the Calibration module as input, and also requires the recipe. It is comprised of two main functions:
**Skill()**: Computes the verification metrics requested in `Workflow:Skill:metric`. The user can request an unlimited number of verification metrics per recipe. The following metrics are currently available:
- `EnsCorr`: Ensemble Mean Correlation.
- `Corr`: Ensemble Correlation.
- `RPS`: Ranked Probability Score.
- `RPSS`: Ranked Probability Skill Score.
- `FRPS`: Fair Ranked Probability Score.
- `FRPSS`: Fair Ranked Probability Skill Score.
- `CRPS`: Continuous Ranked Probability Score.
- `CRPSS`: Continuous Ranked Probability Skill Score.
- `BSS10`: Brier Skill Score (lower extreme).
- `BSS90`: Brier Skill Score (upper extreme).
- `Mean_Bias`: Mean bias of the ensemble.
- `Mean_Bias_SS`: Mean Bias Skill Score.
- `EnsSprErr`: Ensemble Spread-to-Error Ratio.
- `MSSS`: Mean Square Skill Score
- `MSE`: Mean Square Error
**Note**: For the following metrics: "EnsCorr", "FRPS", "RPSS", "FRPSS", "BSS90", "BSS10", if “_specs” is added at the end of the metric (e.g. RPSS_specs), the values will be computed using SpecsVerification.
The output of `Skill()` is a list containing one or more arrays with named dimensions; usually 'var', ‘time’, ‘longitude’ and ‘latitude’. For more details on the specific output for each metric, see the documentation for [s2dv](https://CRAN.R-project.org/package=s2dv) and [SpecsVerification](https://CRAN.R-project.org/package=SpecsVerification).
**Probabilities()** returns a list of two elements with arrays containing the values corresponding to the thresholds in `Workflow:Probabilities:percentiles` (`$percentiles`), as well as their probability bins (`$probs`). Each list contains arrays with named dimensions ‘time’, ‘longitude’ and ‘latitude’.
For example, if the extremes (1/10, 9/10) are requested, the output will be:
`$percentiles`:
- `percentile_10`: The 10th percentile, or lower extreme.
- `percentile_90`: The 90th percentile, or upper extreme.
`$probs`:
- `prob_b10`: Probability of falling below the 10th percentile.
- `prob_10_to_90`: Probability of falling inbetween the 10th and 90th percentile.
- `prob_a90`: Probability of falling above the 90th percentile.
`$probs_fcst`:
- `prob_b10`: Probability of falling below the 10th percentile.
- `prob_10_to_90`: Probability of falling inbetween the 10th and 90th percentile.
- `prob_a90`: Probability of falling above the 90th percentile.
**Note**: When naming the variables, the probability thresholds are converted to percentiles and rounded to the nearest integer to avoid dots in variable or file names. However, this is just a naming convention; the computations are performed based on the original thresholds specified in the recipe.
## Saving
The Saving module contains several functions that export the data (the calibrated hindcast and forecast, the corresponding observations, the skill metrics, percentiles and probabilities) to netCDF files and save them.
Since version 1.x.x of SUNSET, each module has different options to save all or some the data produced. These options can be specified in the recipe. To see the options for a specific module, consult the 'Recipe' section of this wiki or the documentation for that module. Once the computations are finished, the output subdirectories and files are generated internally within the module function, according to the user's specifications.
Inside the main output directory, the netCDF files produced by each module will be stored inside `/outputs/<module_name>/`. The old function `save_data()`, which has been renamed to `Saving()`, still exists, but will likely be deprecated in the near future.
### The structure of the output directory ### The structure of the output directory
The outputs are saved to a unique folder inside the directory you specified in the recipe, under outputs/. The structure of the subdirectories and file names will depend on the option you specify for '`Output_format`' in the recipe. The outputs are saved to a unique folder inside the directory you specified in the recipe, under `outputs/`. For example, if you are saving the metrics computed with the Skill module, they will be stored inside `<output_dir>/outputs/Skill/`, where <output_dir> is the output directory specified in the recipe.
The structure of the subdirectories and file names will depend on the selected '`Output_format`' in the recipe.
There are two possible output formats currently: 'S2S4E' and 'Scorecards', described below. To request the inclusion of an additional output format, please open an issue. There are two possible output formats currently: `'S2S4E'` and `'Scorecards'`, described below. To request the inclusion of an additional output format, please open an issue. It is also possible to save the data to a custom directory in a custom format, by making use of the CSTools functions `SaveExp()` and `CST_SaveExp()`.
#### S2S4E (default) #### S2S4E (default)
With the 'S2S4E' output format, the structure of the subdirectory and the files is as follows: With the 'S2S4E' output format, the structure of the subdirectory and the files is as follows:
If fcst_year has been requested: If fcst_year has been requested:
- `output_dir/outputs/<system>/<calibration_method>-<frequency>/<forecast_date>/<var>/` - `<system>/<calibration_method>-<frequency>/<forecast_date>/<var>/`
If fcst_year is empty: If fcst_year is empty:
- For seasonal data: `output_dir/outputs/<system>/<calibration_method>-<frequency>/hcst-<mmdd>/<var>/` - For seasonal data: `<system>/<calibration_method>-<frequency>/hcst-<mmdd>/<var>/`
- For decadal data: `output_dir/outputs/<system>/<calibration_method>-<frequency>/hcst-<yyyy>_<yyyy>/<var>/` - For decadal data: `<system>/<calibration_method>-<frequency>/hcst-<yyyy>_<yyyy>/<var>/`
For example, in our example recipe, the final output directory will be: For example, in our example recipe, the calibrated tas netCDF Files will be in:
`/esarchive/scratch/vagudets/repos/sunset/out-logs/recipe-wiki_20221025164151/outputs/mse_min-monthly_mean/20201101/tas/` `/esarchive/scratch/vagudets/repos/sunset-outputs/recipe-wiki_20221025164151/outputs/Calibration/mse_min-monthly_mean/20201101/tas/`
The structure of the names of the files is: The structure of the names of the files is:
- Skill metrics: `<var>-skill_month<mm>.nc` - Skill metrics: `<var>-skill_month<mm>.nc`
...@@ -302,13 +742,33 @@ For the 'Scorecards' output format, the structure of the subdirectory and the fi ...@@ -302,13 +742,33 @@ For the 'Scorecards' output format, the structure of the subdirectory and the fi
`system` and `reference`are abbreviations of the names of the system and reference, respectively. `system` and `reference`are abbreviations of the names of the system and reference, respectively.
`var` is the name of the variable, `startyear` and `endyear` are the initial and final hindcast years, `yyyymmdd` is the start date for each year and `mm` is the initialization month. `var` is the name of the variable, `startyear` and `endyear` are the initial and final hindcast years, `yyyymmdd` is the start date for each year and `mm` is the initialization month.
### The Saving() wrapper function (to be deprecated)
`Saving()` served as the main wrapper function for this module. This function will be deprecated in a future release.
The main arguments are `recipe` (the recipe) and `data` (the list of `s2dv_cube` objects containing at least the processed hindcast and observation data). The rest of the arguments (skill_metrics and probabilities) are optional. Including `skill_metrics` and `probabilities` will save the skill metrics and the percentiles and probability bins, respectively.
```R
# Therefore, all the data can be saved at once:
Saving(recipe, data, skill_metrics, probabilities)
# Or, one can choose to save only some of it. For example,
# saving the calibrated data and corresponding observations only:
Saving(recipe, data)
# Or saving the hcst/fcst/obs and skill metrics only:
Saving(recipe, data, skill_metrics = skill_metrics)
# Or only hcst/fcst/obs and the probabilities:
Saving(recipe, data, probabilities = probabilities)
```
## Visualization module ## Visualization module
The Visualization module provides a few basic plots to visualize the data loaded and computed using the previous modules. The color palettes and titles of the plots will vary depending on the output format requested in the recipe. The palettes are generated using the clim.colors() function from the s2dv package. The Visualization module provides a few basic plots to visualize the data loaded and computed using the previous modules. The color palettes and titles of the plots will vary depending on the output format requested in the recipe. The palettes are generated using the clim.colors() function from the [s2dv](https://cran.r-project.org/web/packages/s2dv/index.html) package.
- For 'S2S4E', the "bluered" palette is used. - For 'S2S4E', the "bluered" palette is used.
- For 'Scorecards', "purpleorange" is used. - For 'Scorecards', "purpleorange" is used.
**plot_data()** is the main wrapper function for this module. It generates the plots and saves them to the same output directory described in the Saving module section, under `plots/`. **Visualization()** is the main wrapper function for this module. It generates the plots and saves them to the same output directory described in the Saving module section, under `plots/`.
This function's parameters are similar to those in the Saving module: This function's parameters are similar to those in the Saving module:
...@@ -317,18 +777,20 @@ This function's parameters are similar to those in the Saving module: ...@@ -317,18 +777,20 @@ This function's parameters are similar to those in the Saving module:
- `probabilities`: List in the format of the Skill module output, containing the 33rd and 66th percentiles. - `probabilities`: List in the format of the Skill module output, containing the 33rd and 66th percentiles.
- If `significance = TRUE` (default), the statistical significance dots will be displayed in the plot, when available. It defaults to FALSE. - If `significance = TRUE` (default), the statistical significance dots will be displayed in the plot, when available. It defaults to FALSE.
`plot_data()` attempts to generate: `Visualization()` attempts to generate:
- Plots of all the skill metrics provided in `skill_metrics`. - Plots of all the skill metrics provided in `skill_metrics`.
- A plot of the forecast ensemble mean, if a forecast has been provided. - A plot of the forecast ensemble mean, if a forecast has been provided.
- A Most Likely Terciles plot, if the `probabilities` include the terciles (percentiles 33 and 66). - A Most Likely Terciles plot, if the `probabilities` include the terciles (percentiles 33 and 66).
The three functions that `plot_data()` calls can also be called independently. In this case, archive configuration file and the the output directory for the plots will have to be provided explicitly: The three functions that `Visualization()` calls can also be called independently. In this case, archive configuration file and the the output directory for the plots will have to be provided explicitly as arguments to :
**plot_skill_metrics(recipe, archive, data_cube, skill_metrics, outdir, significance = F)**: Generates, for each metric in skill_metrics, a figure with one plot per time step, and saves each figure to the output directory `outdir` as `<metric>.png`. **plot_skill_metrics(recipe, archive, data_cube, skill_metrics, outdir, significance = F)**: Generates, for each metric in skill_metrics, a figure with one plot per time step, and saves each figure to the output directory `outdir` as `<metric>.png`.
`data_cube` is an s2dv_cube containing the appropriate metadata, for example the hcst object from the Loading module. `data_cube` is an s2dv_cube containing the appropriate metadata, for example the hcst object that was passed to the Skill module.
**plot_ensemble_mean(recipe, archive, fcst, outdir)**: Computes the forecast ensemble mean and generates a figure with one plot per time step, and saves it to the output directory `outdir` as `forecast_ensemble_mean.png` **plot_ensemble_mean(recipe, archive, fcst, outdir)**: Computes the forecast ensemble mean and generates a figure with one plot per time step, and saves it to the output directory `outdir` as `forecast_ensemble_mean.png`
**plot_most_likely_terciles(recipe, archive, fcst, percentiles, outdir)**: Computes the forecast tercile probability bins with respect to the terciles provided in 'percentiles', then generates a figure with one plot per time step and saves it to the directory `outdir` as `forecast_most_likely_terciles.png` **plot_most_likely_terciles(recipe, archive, fcst, percentiles, outdir)**: Computes the forecast tercile probability bins with respect to the terciles provided in 'percentiles', then generates a figure with one plot per time step and saves it to the directory `outdir` as `forecast_most_likely_terciles.png`
\ No newline at end of file