|
|
|
|
|
## Introduction
|
|
|
|
|
|
This the GitLab repository for the SUNSET (SUbseasoNal to decadal climate forecast post-processIng and asSEmenT suite), a modular tool for subseasonal to seasonal to decadal forecast verification workflows. It is intended to have a modularized structure, where each module is a separate part of the code that performs a specific task, so that parts of the workflow can be skipped or reordered.
|
|
|
|
|
|
The datasets, forecast horizon, time period, skill metrics to compute and other parameters are specified by the user in a configuration file, called "recipe".
|
|
|
|
|
|
After definining the recipe, the user can create a script using the functions available in the tool, which use apply the information in the recipe to do the desired computations. You an example script [in the SUNSET code Snippets](https://earth.bsc.es/gitlab/es/sunset/-/snippets/111).
|
|
|
|
|
|
- Modules currently available: Loading, Calibration, Anomalies, Skill, Indices, Saving, Visualization
|
|
|
- Modules in development: Downscaling, Scorecards
|
|
|
- Future modules: Aggregation, Indicators
|
|
|
|
|
|
This tool is in the early stages of development, so the code and the information in this wiki may be subject to frequent changes and updates. This wiki contains all the information needed to use the available modules.
|
|
|
|
|
|
## Recipes
|
|
|
|
|
|
In order to use SUNSET, users must define a recipe containing all the information pertaining to their workflow.
|
|
|
|
|
|
Here is an example of a recipe to load monthly mean ECMWF System 5 data from `/esarchive/`, with a 1993 to 2016 hindcast period, the corresponding ERA5 observations, and a 2020 forecast for the November initialization, for the months of November and December.
|
|
|
|
|
|
Following this recipe, the observations will be interpolated to the experiment grid (Regrid type: 'to_system') using bilinear interpolation. The hindcast and forecast will be calibrated using Quantile Mapping, and the Ranked Probability Skill Score (RPSS) and Continuous Ranked Probability Skill Score (CRPSS) will be computed.
|
|
|
The terciles (1/3, 2/3), quartiles (1/4, 2/4, 3/4), extremes (1/10, 9/10) and their corresponding probability bins will also be computed. Any output files will be saved inside the output directory.
|
|
|
|
|
|
```yaml
|
|
|
Description:
|
|
|
Author: V. Agudetse
|
|
|
Info: ECMWF System5 Seasonal Forecast Example recipe (monthly mean, tas)
|
|
|
|
|
|
Analysis:
|
|
|
Horizon: seasonal # Mandatory, str: 'subseasonal', 'seasonal', or 'decadal'
|
|
|
Variables:
|
|
|
# name: variable name(s) in the /esarchive (Mandatory, str)
|
|
|
# freq: 'monthly_mean', 'daily' or 'daily_mean' (Mandatory, str)
|
|
|
- {name: 'tas', freq: 'monthly_mean', units: 'C'}
|
|
|
Datasets:
|
|
|
System:
|
|
|
# name: System name (Mandatory, str)
|
|
|
# member: 'all' or individual members, separated by a comma and in quotes (decadal only, str)
|
|
|
- {name: 'ECMWF-SEAS5', member: 'all'}
|
|
|
Multimodel: no # Either yes/true or no/false (Mandatory, bool)
|
|
|
Reference:
|
|
|
- {name: 'ERA5'} # Reference name (Mandatory, str)
|
|
|
Time:
|
|
|
sdate: '1101' # Start date, 'mmdd' (Mandatory, int)
|
|
|
fcst_year: '2020' # Forecast initialization year 'YYYY' (Optional, int)
|
|
|
hcst_start: '1993' # Hindcast initialization start year 'YYYY' (Mandatory, int)
|
|
|
hcst_end: '2016' # Hindcast initialization end year 'YYYY' (Mandatory, int)
|
|
|
ftime_min: 1 # First forecast time step in months. Starts at “1”. (Mandatory, int)
|
|
|
ftime_max: 6 # Last forecast time step in months. Starts at “1”. (Mandatory, int)
|
|
|
Region:
|
|
|
latmin: -90 # minimum latitude (Mandatory, int)
|
|
|
latmax: 90 # maximum latitude (Mandatory, int)
|
|
|
lonmin: 0 # minimum longitude (Mandatory, int)
|
|
|
lonmax: 359.9 # maximum longitude (Mandatory, int)
|
|
|
Regrid:
|
|
|
method: bilinear # Interpolation method (Mandatory, str)
|
|
|
type: to_system # Interpolate to: 'to_system', 'to_reference', 'none',
|
|
|
# or CDO-accepted grid. (Mandatory, str)
|
|
|
Workflow:
|
|
|
Calibration:
|
|
|
method: raw # Calibration method. (Mandatory, str)
|
|
|
save: 'none' # Options: 'all', 'none', 'exp_only', 'fcst_only' (Mandatory, str)
|
|
|
Anomalies:
|
|
|
compute: no # Either yes/true or no/false (Mandatory, bool)
|
|
|
cross_validation: no # Either yes/true or no/false (Mandatory if 'compute: yes', bool)
|
|
|
save: 'fcst_only' # Options: 'all', 'none', 'exp_only', 'fcst_only' (Mandatory, str)
|
|
|
Downscaling:
|
|
|
# Assumption 1: leave-one-out cross-validation is always applied
|
|
|
# Assumption 2: for analogs, we select the best analog (minimum distance)
|
|
|
type: intbc # mandatory, 'none', 'int', 'intbc', 'intlr', 'analogs', 'logreg'.
|
|
|
int_method: conservative # regridding method accepted by CDO.
|
|
|
bc_method: bias # If type intbc. Options: 'bias', 'calibration', 'quantile_mapping', 'qm', 'evmos', 'mse_min', 'crps_min', 'rpc-based'.
|
|
|
lr_method: # If type intlr. Options: 'basic', 'large_scale', '4nn'
|
|
|
log_reg_method: # If type logreg. Options: 'ens_mean', 'ens_mean_sd', 'sorted_members'
|
|
|
target_grid: /esarchive/recon/ecmwf/era5/monthly_mean/tas_f1h/tas_200002.nc # nc file or grid accepted by CDO
|
|
|
nanalogs: # If type analgs. Number of analogs to be searched
|
|
|
save: 'all' # 'all'/'none'/'exp_only'
|
|
|
Indices:
|
|
|
## Indices available: NAO, Nino1+2, Nino3, Nino3.4, Nino4.
|
|
|
## Each index can only be computed if its area is within the selected region.
|
|
|
# obsproj: NAO computation method (see s2dv::NAO()) Default is yes/true. (Optional, bool)
|
|
|
# save: What to save. Options: 'all'/'none'. Default is 'all'.
|
|
|
# plot_ts: Generate time series plot? Default is yes/true. (Optional, bool)
|
|
|
# plot_sp: Generate spatial pattern plot? Default is yes/true. (Optional, bool)
|
|
|
NAO: {obsproj: yes, save: 'all', plot_ts: yes, plot_sp: yes}
|
|
|
Nino1+2: {save: 'all', plot_ts: yes, plot_sp: yes}
|
|
|
Nino3: {save: 'all', plot_ts: yes, plot_sp: yes}
|
|
|
Nino3.4: {save: 'all', plot_ts: yes, plot_sp: yes}
|
|
|
Nino4: {save: 'all', plot_ts: yes, plot_sp: yes}
|
|
|
Skill:
|
|
|
metric: RPSS CRPSS # List of skill metrics separated by spaces or commas. (Mandatory, str)
|
|
|
save: 'all' # Options: 'all', 'none' (Mandatory, str)
|
|
|
Probabilities:
|
|
|
percentiles: [[1/3, 2/3], [1/10, 9/10], [1/4, 2/4, 3/4]] # Thresholds
|
|
|
# for quantiles and probability categories. Each set of thresholds should be
|
|
|
# enclosed within brackets. (Optional)
|
|
|
save: 'percentiles_only' # Options: 'all', 'none', 'bins_only', 'percentiles_only' (Mandatory, str)
|
|
|
Visualization:
|
|
|
plots: skill_metrics, most_likely_terciles, forecast_ensemble_mean # Types of plots to generate (Optional, str)
|
|
|
multi_panel: yes # Multi-panel plot or single-panel plots. Default is 'no/false'. (Optional, bool)
|
|
|
projection: 'cylindrical_equidistant' # Options: 'cylindrical_equidistant', 'robinson', 'lambert_europe'. Default is cylindrical equidistant. (Optional, str)
|
|
|
mask_terciles: no # Whether to mask the non-significant points by rpss in the most likely tercile plot. yes/true, no/false or 'both'. Default is no/false. (Optional, str)
|
|
|
dots_terciles: yes # Whether to dot the non-significant by rpss in the most likely tercile plot. yes/true, no/false or 'both'. Default is no/false. (Optional, str)
|
|
|
Indicators:
|
|
|
index: no # This feature is not implemented yet
|
|
|
ncores: 10 # Number of cores to be used in parallel computation.
|
|
|
# If left empty, defaults to 1. (Optional, int)
|
|
|
remove_NAs: yes # Whether to remove NAs.
|
|
|
# If left empty, defaults to no/false. (Optional, bool)
|
|
|
Output_format: 'S2S4E' # 'S2S4E' or 'Scorecards'. Determines the format of the output. Default is 'S2S4E'.
|
|
|
Run:
|
|
|
Loglevel: INFO # Minimum category of log messages to display: 'DEBUG', 'INFO', 'WARN', 'ERROR' or 'FATAL'.
|
|
|
# Default value is 'INFO'. (Optional, str)
|
|
|
Terminal: yes # Optional, bool: Whether to display log messages in the terminal.
|
|
|
# Default is yes/true.
|
|
|
output_dir: /esarchive/scratch/vagudets/repos/sunset-outputs/ # Output directory. Must have write permissions. (Mandatory, str)
|
|
|
code_dir: /esarchive/scratch/vagudets/repos/sunset/
|
|
|
```
|
|
|
|
|
|
## List of /esarchive/ datasets
|
|
|
|
|
|
Here is a list of the datasets that can currently be loaded by the tool. To request that an additional dataset be added, please [open an issue](https://earth.bsc.es/gitlab/es/sunset/-/issues).
|
|
|
|
|
|
### Seasonal datasets
|
|
|
|
|
|
Systems:
|
|
|
| Forecast System | Monthly | Daily | /esarchive/exp/ path |
|
|
|
|----------------------------|-----------------|------------|-------------------------|
|
|
|
| **ECMWF-SEAS5** | Yes | Yes | ecmwf/system5c3s/ |
|
|
|
| **ECMWF-SEAS5.1** | Yes | Yes | ecmwf/system51c3s/ |
|
|
|
| **DWD-GCFS2.1** | Yes | No | dwd/system21_m1/ |
|
|
|
| **CMCC-SPS3.5** | Yes | No | cmcc/system35c3s/ |
|
|
|
| **Meteo-France-System7** | Yes | No | meteofrance/system7c3s/ |
|
|
|
| **JMA-CPS2** | Yes | No | jma/system2c3s/ |
|
|
|
| **ECCC-CanCM4i** | May to November | No | eccc/eccc1/ |
|
|
|
| **UK-MetOffice-GloSea600** | Yes | No | ukmo/glosea6_system600-c3s/ |
|
|
|
| **NCEP-CFSv2** | Yes | No | ncep/cfs-v2/ |
|
|
|
|
|
|
Observations:
|
|
|
| Reference | Monthly | Daily | /esarchive/recon/ path |
|
|
|
|---------------|--------------|------------|------------------------|
|
|
|
| **ERA5** | Yes | Yes | ecmwf/era5/ |
|
|
|
| **ERA5-Land** | `tas` only | Yes | ecmwf/era5land/ |
|
|
|
| **UERRA** | No | `tas` only | ecmwf/uerra_mescan/ |
|
|
|
|
|
|
### Decadal datasets
|
|
|
|
|
|
Systems:
|
|
|
| Forecast System | Monthly | Daily | forecast (DCPP-B)? |
|
|
|
|------------------------|--------------|------------|--------------------|
|
|
|
| **BCC-CSM2-MR** | Yes | Yes | No |
|
|
|
| **CanESM5** | Yes | Yes | Yes |
|
|
|
| **CESM1-1-CAM5-CMIP5** | Yes | WIP | No |
|
|
|
| **CMCC-CM2-SR5** | Yes | Yes | Yes |
|
|
|
| **EC-Earth3-i1** | Yes | Yes | No |
|
|
|
| **EC-Earth3-i2** | Yes | Yes | No |
|
|
|
| **EC-Earth3-i4** | Yes | Yes | Yes |
|
|
|
| **HadGEM3-GC3.1-MM** | Yes | Yes | Yes |
|
|
|
| **IPSL-CM6A-LR** | Yes | Yes | No |
|
|
|
| **MIROC6** | Yes | Yes | No |
|
|
|
| **MPI-ESM1.2-HR** | Yes | Yes | No |
|
|
|
| **MPI-ESM1.2-LR** | Yes | No | Yes |
|
|
|
| **MRI-ESM2-0** | Yes | Yes | No |
|
|
|
| **NorCPM1-i1** | Yes | Yes | No |
|
|
|
| **NorCPM1-i2** | Yes | Yes | No |
|
|
|
|
|
|
Observations:
|
|
|
| Reference | Monthly | Daily |
|
|
|
|---------------|--------------|------------|
|
|
|
| **GHCNv4** | Yes | No |
|
|
|
| **ERA5** | Yes | Yes |
|
|
|
| **JRA-55** | Yes | Yes |
|
|
|
| **GISTEMPv4** | Yes | No |
|
|
|
| **HadCRUT4** | Yes | No |
|
|
|
| **HadSLP2** | Yes | No |
|
|
|
|
|
|
## The first steps
|
|
|
|
|
|
In order to run the SUNSET Suite, you need to load the necessary modules (R, CDO, GDAL, GEOS and PROJ). To do this, you can run the command `source MODULES` in the terminal, from the main folder of the SUNSET repository.
|
|
|
|
|
|
Before calling the modules in your script or in the R console, you should run the `prepare_outputs()` function as shown in the [example script](https://earth.bsc.es/gitlab/es/sunset/-/snippets/96), which will read your recipe and set up the directory for your outputs.
|
|
|
`prepare_outputs()` will perform a check on your recipe to detect potential errors. If you want to disable this check, you may set the argument `disable_checks = TRUE` when calling the function.
|
|
|
|
|
|
If you had a recipe named 'recipe-wiki.yml' that looked like the example in this wiki, this directory might look something like this:
|
|
|
|
|
|
`/esarchive/scratch/vagudets/repos/sunset-outputs/recipe-wiki_20221025164151/`
|
|
|
|
|
|
Inside you will find a log file, a copy of your recipe, and your outputs from the Saving and Visualization modules.
|
|
|
|
|
|
## Loading module
|
|
|
|
|
|
The Loading module retrieves the data requested in the recipe from /esarchive/, interpolates it to the desired grids if interpolation has been requested, and converts it to objects of class `s2dv_cube`, which can be passed onto the other modules in the tool. An `s2dv_cube` object is a list containing the data array in the element $data and many other elements that store the metadata.
|
|
|
|
|
|
The output of the main function, `Loading()`, is a list containing the hindcast, observations and forecast, named hcst, obs and fcst respectively. fcst will be `NULL` if no forecast years have been requested.
|
|
|
|
|
|
### Regridding
|
|
|
|
|
|
The Loading module can interpolate the data while loading them, using CDO as the core tool. The interpolation methods that can be specified in the recipe under `Regrid:method` are those accepted by CDO: `'conservative'`, `'bilinear'`,`'bicubic'`, `'distance-weighted'`, `'con2'`, `'laf'` and `'nn'`. Consult the CDO User Guide for more details.
|
|
|
|
|
|
The target grid is to be specified in `Regrid:type`. The options are:
|
|
|
- `‘to_system’`: The observations are interpolated to the system grid.
|
|
|
- `‘to_reference’:` the hindcast and forecast are interpolated to the reference grid.
|
|
|
- `‘none’`: No interpolation is done when loading the data.
|
|
|
- A CDO-accepted grid format, such as `'r360x180'` or the link to a netCDF file. In this case, both the system and the reference will be interpolated to this grid. See the FAQs section for more details.
|
|
|
|
|
|
### Loading the desired latitude and longitude range
|
|
|
|
|
|
The Verification Suite uses [startR](https://CRAN.R-project.org/package=startR) to load the data, which you may need to take into account to make sure you can load your desired region or range. For example, setting `lonmin: 0` and `lonmax: 360` can be problematic because the `Start()` function will consider them the same point.
|
|
|
|
|
|
The startR documentation has [a guide to explain how to select your longitude range appropriately](https://earth.bsc.es/gitlab/es/startR/-/blob/master/inst/doc/faq.md#11-select-the-longitudelatitude-region). The Loading module handles the `Sort()` and `CircularSort()` automatically from the ranges in your recipe.
|
|
|
|
|
|
## Units module
|
|
|
|
|
|
The Units module should be used by calling the Relocation leaveUnits() function after the Loading module to ensure consistency between system and reference datasets. It will verify that the hindcast, forecast and observations share the same units, and make the appropriate transformations if there are any differences.
|
|
|
|
|
|
Additionally, user-specified units can be included in the recipe for each variable. Currently, the units of three variables can be converted. The following units are considered:
|
|
|
- **Temperature:** `"C", "K"`
|
|
|
- **Precipitation:** `"ms-1", "kgm-2s-1", "mm", "m"` For the precipitation units, the user can also decide whether to obtain the total amount of the flux by setting parameter`flux` to true/false in the Variable section of the recipe.
|
|
|
- **Pressure:** `"pa", "hpa", "mb"`
|
|
|
|
|
|
## Calibration module
|
|
|
|
|
|
The Calibration module performs bias correction on the loaded data. It accepts the output of the Loading module as input, and also requires the recipe. It applies the selected bias correction method to the hindcast and forecast data using the observations as a reference, and returns the calibrated data and its metadata as an `s2dv_cube` object.
|
|
|
|
|
|
The output of the main function, `Calibration()`, is a list containing the calibrated hindcast and forecast, named hcst and fcst respectively. fcst will be `NULL` if no forecast years have been requested.
|
|
|
|
|
|
### Calibration methods currently available:
|
|
|
|
|
|
The calibration method can be requested in the `Workflow:Calibration:method` section of the recipe. **The user can only request one calibration method per recipe.** This is a list of the methods currently available:
|
|
|
|
|
|
- `'raw'`: No calibration is performed. A warning will show up on the terminal when `Calibration()` is called, and it will return the uncalibrated hindcast and forecast.
|
|
|
|
|
|
- Daily data: Quantile mapping `'qmap'`.
|
|
|
For more details, see the [CSTools documentation](https://CRAN.R-project.org/package=CSTools) for `CST_QuantileMapping()`.
|
|
|
|
|
|
- Monthly data: `'bias'`, `'evmos'`, `'mse_min'`, `'crps_min'`, and `'rpc-based'`.
|
|
|
For more details, see the [CSTools documentation](https://CRAN.R-project.org/package=CSTools) for `CST_Calibration()`.
|
|
|
|
|
|
## Anomalies module
|
|
|
|
|
|
The Anomalies module computes the anomalies of the data with respect to the climatological mean; with or without cross-validation, depending on what it specified in the recipe. It accepts the output of either the Loading or the Calibration module as input, and also requires the recipe. It makes use of the CSTools function `CST_Anomaly()`.
|
|
|
|
|
|
The output of the main function, `Anomalies()`, is a list of `s2dv_cube` objects containing the anomalies for the hcst, fcst and obs, as well as the original hcst and obs full fields in case they are needed for later computations.
|
|
|
|
|
|
## Indices module
|
|
|
|
|
|
The Indices module aggregates the hindcast and reference data to compute climatological indices such as the North Atlantic Oscillation (NAO) or El Niño indices.
|
|
|
The main function, `Indices()`, returns the hcst and obs s2dv_cube objects for each requested index, in the form of a list of lists. The 'latitude' and 'longitude' dimensions of the original arrays are aggregated into a single 'region' dimension.
|
|
|
|
|
|
Indices currently available:
|
|
|
|
|
|
| Index | Recipe name | Longitude range | Latitude range |
|
|
|
|--------------------------------|---------------|-------------------|------------------|
|
|
|
| **North Atlantic Oscillation** | NAO | -80º to 40º | 20º to 80º |
|
|
|
| **Niño 1+2** | Nino1+2 | -90º to -80º | -10º to 0º |
|
|
|
| **Niño 3** | Nino3 | -150º to -90º | -5º to 5º |
|
|
|
| **Niño 3.4** | Nino3.4 | -170º to -120º | -5º to 5º |
|
|
|
| **Niño 4** | Nino4 | 160º to -150º | -5º to 5º |
|
|
|
|
|
|
|
|
|
## Skill module
|
|
|
|
|
|
The Skill module is the part of the workflow that computes the metrics to assess the quality of a forecast. It accepts the output of the Calibration module as input, and also requires the recipe. It is comprised of two main functions:
|
|
|
|
|
|
**Skill()**: Computes the verification metrics requested in `Workflow:Skill:metric`. The user can request an unlimited number of verification metrics per recipe. The following metrics are currently available:
|
|
|
|
|
|
- `EnsCorr`: Ensemble Mean Correlation.
|
|
|
- `Corr`: Ensemble Correlation.
|
|
|
- `RPS`: Ranked Probability Score.
|
|
|
- `RPSS`: Ranked Probability Skill Score.
|
|
|
- `FRPS`: Fair Ranked Probability Score.
|
|
|
- `FRPSS`: Fair Ranked Probability Skill Score.
|
|
|
- `CRPS`: Continuous Ranked Probability Score.
|
|
|
- `CRPSS`: Continuous Ranked Probability Skill Score.
|
|
|
- `BSS10`: Brier Skill Score (lower extreme).
|
|
|
- `BSS90`: Brier Skill Score (upper extreme).
|
|
|
- `Mean_Bias`: Mean bias of the ensemble.
|
|
|
- `Mean_Bias_SS`: Mean Bias Skill Score.
|
|
|
- `EnsSprErr`: Ensemble Spread-to-Error Ratio.
|
|
|
- `MSSS`: Mean Square Skill Score
|
|
|
- `MSE`: Mean Square Error
|
|
|
|
|
|
**Note**: For the following metrics: "EnsCorr", "FRPS", "RPSS", "FRPSS", "BSS90", "BSS10", if “_specs” is added at the end of the metric (e.g. RPSS_specs), the values will be computed using SpecsVerification.
|
|
|
|
|
|
The output of `Skill()` is a list containing one or more arrays with named dimensions; usually 'var', ‘time’, ‘longitude’ and ‘latitude’. For more details on the specific output for each metric, see the documentation for [s2dv](https://CRAN.R-project.org/package=s2dv) and [SpecsVerification](https://CRAN.R-project.org/package=SpecsVerification).
|
|
|
|
|
|
**Probabilities()** returns a list of two elements with arrays containing the values corresponding to the thresholds in `Workflow:Probabilities:percentiles` (`$percentiles`), as well as their probability bins (`$probs`). Each list contains arrays with named dimensions ‘time’, ‘longitude’ and ‘latitude’.
|
|
|
For example, if the extremes (1/10, 9/10) are requested, the output will be:
|
|
|
|
|
|
`$percentiles`:
|
|
|
- `percentile_10`: The 10th percentile, or lower extreme.
|
|
|
- `percentile_90`: The 90th percentile, or upper extreme.
|
|
|
|
|
|
`$probs`:
|
|
|
- `prob_b10`: Probability of falling below the 10th percentile.
|
|
|
- `prob_10_to_90`: Probability of falling inbetween the 10th and 90th percentile.
|
|
|
- `prob_a90`: Probability of falling above the 90th percentile.
|
|
|
|
|
|
`$probs_fcst`:
|
|
|
- `prob_b10`: Probability of falling below the 10th percentile.
|
|
|
- `prob_10_to_90`: Probability of falling inbetween the 10th and 90th percentile.
|
|
|
- `prob_a90`: Probability of falling above the 90th percentile.
|
|
|
|
|
|
**Note**: When naming the variables, the probability thresholds are converted to percentiles and rounded to the nearest integer to avoid dots in variable or file names. However, this is just a naming convention; the computations are performed based on the original thresholds specified in the recipe.
|
|
|
|
|
|
## Saving
|
|
|
|
|
|
The Saving module contains several functions that export the data (the calibrated hindcast and forecast, the corresponding observations, the skill metrics, percentiles and probabilities) to netCDF files and save them.
|
|
|
|
|
|
Since version 1.x.x of SUNSET, each module has different options to save all or some the data produced. These options can be specified in the recipe. To see the options for a specific module, consult the 'Recipe' section of this wiki or the documentation for that module. Once the computations are finished, the output subdirectories and files are generated internally within the module function, according to the user's specifications.
|
|
|
|
|
|
Inside the main output directory, the netCDF files produced by each module will be stored inside `/outputs/<module_name>/`. The old function `save_data()`, which has been renamed to `Saving()`, still exists, but will likely be deprecated in the near future.
|
|
|
|
|
|
### The structure of the output directory
|
|
|
|
|
|
The outputs are saved to a unique folder inside the directory you specified in the recipe, under `outputs/`. For example, if you are saving the metrics computed with the Skill module, they will be stored inside `<output_dir>/outputs/Skill/`, where <output_dir> is the output directory specified in the recipe.
|
|
|
|
|
|
The structure of the subdirectories and file names will depend on the selected '`Output_format`' in the recipe.
|
|
|
|
|
|
There are two possible output formats currently: `'S2S4E'` and `'Scorecards'`, described below. To request the inclusion of an additional output format, please open an issue. It is also possible to save the data to a custom directory in a custom format, by making use of the CSTools functions `SaveExp()` and `CST_SaveExp()`.
|
|
|
|
|
|
#### S2S4E (default)
|
|
|
With the 'S2S4E' output format, the structure of the subdirectory and the files is as follows:
|
|
|
|
|
|
If fcst_year has been requested:
|
|
|
- `<system>/<calibration_method>-<frequency>/<forecast_date>/<var>/`
|
|
|
|
|
|
If fcst_year is empty:
|
|
|
- For seasonal data: `<system>/<calibration_method>-<frequency>/hcst-<mmdd>/<var>/`
|
|
|
- For decadal data: `<system>/<calibration_method>-<frequency>/hcst-<yyyy>_<yyyy>/<var>/`
|
|
|
|
|
|
For example, in our example recipe, the calibrated tas netCDF Files will be in:
|
|
|
|
|
|
`/esarchive/scratch/vagudets/repos/sunset-outputs/recipe-wiki_20221025164151/outputs/Calibration/mse_min-monthly_mean/20201101/tas/`
|
|
|
|
|
|
The structure of the names of the files is:
|
|
|
- Skill metrics: `<var>-skill_month<mm>.nc`
|
|
|
- Correlation: `<var>-corr_month<mm>.nc`
|
|
|
- Processed hindcast and forecast data: `<var>_<yyyymmdd>.nc`
|
|
|
- Observations: `<var>-obs_<yyyymmdd>.nc`
|
|
|
- Probabilities: `<var>-probs_<yyyymmdd>.nc`
|
|
|
- Percentiles: `<var>-percentiles_month<mm>.nc`
|
|
|
|
|
|
Where `var` is the name of the variable, `yyyy` is the year, `mm` is the initialization month and `mm` is the initialization day.
|
|
|
|
|
|
#### Scorecards:
|
|
|
For the 'Scorecards' output format, the structure of the subdirectory and the files is:
|
|
|
|
|
|
- Skill metrics: `<system>/<variable>/scorecards_<system>-<reference>_<variable>-skill_<startyear>_<endyear>_s<mm>.nc`
|
|
|
- Processed hindcast and forecast data: `<system>/<variable>/scorecards_<system>-<reference>_<variable>_<yyyymmdd>_<startyear>_<endyear>_s<mm>.nc`
|
|
|
- Observations: `<system>/<variable>/scorecards_<system>-<reference>_<variable>-obs_<yyyymmdd>_<startyear>_<endyear>_s<mm>.nc`
|
|
|
- Probabilities: `<system>/<variable>/scorecards_<system>-<reference>_<variable>-probs_<yyyymmdd>_<startyear>_<endyear>_s<mm>.nc`
|
|
|
- Percentiles: `<system>/<variable>/scorecards_<system>-<reference>_<variable>-percentiles_<startyear>_<endyear>_s<mm>.nc`
|
|
|
|
|
|
`system` and `reference`are abbreviations of the names of the system and reference, respectively.
|
|
|
`var` is the name of the variable, `startyear` and `endyear` are the initial and final hindcast years, `yyyymmdd` is the start date for each year and `mm` is the initialization month.
|
|
|
|
|
|
### The Saving() wrapper function (to be deprecated)
|
|
|
|
|
|
`Saving()` serves as the main wrapper function for this module.`recipe` (the recipe) and `data` (the list of `s2dv_cube` objects containing at least the processed hindcast and observation data). The rest of the arguments (skill_metrics and probabilities) are optional. Including `skill_metrics` and `probabilities` will save the skill metrics and the percentiles and probability bins, respectively.
|
|
|
|
|
|
```R
|
|
|
# Therefore, all the data can be saved at once:
|
|
|
Saving(recipe, data, skill_metrics, probabilities)
|
|
|
|
|
|
# Or, one can choose to save only some of it. For example,
|
|
|
# saving the calibrated data and corresponding observations only:
|
|
|
Saving(recipe, data)
|
|
|
|
|
|
# Or saving the hcst/fcst/obs and skill metrics only:
|
|
|
Saving(recipe, data, skill_metrics = skill_metrics)
|
|
|
|
|
|
# Or only hcst/fcst/obs and the probabilities:
|
|
|
Saving(recipe, data, probabilities = probabilities)
|
|
|
```
|
|
|
|
|
|
## Visualization module
|
|
|
|
|
|
The Visualization module provides a few basic plots to visualize the data loaded and computed using the previous modules. The color palettes and titles of the plots will vary depending on the output format requested in the recipe. The palettes are generated using the clim.colors() function from the [s2dv](https://cran.r-project.org/web/packages/s2dv/index.html) package.
|
|
|
- For 'S2S4E', the "bluered" palette is used.
|
|
|
- For 'Scorecards', "purpleorange" is used.
|
|
|
|
|
|
**Visualization()** is the main wrapper function for this module. It generates the plots and saves them to the same output directory described in the Saving module section, under `plots/`.
|
|
|
|
|
|
This function's parameters are similar to those in the Saving module:
|
|
|
|
|
|
`recipe` (the recipe) and `data` (the list obtained from the Loading module) are mandatory arguments. The rest of the arguments (`skill_metrics`, `probabilities`, and `significance`) are optional:
|
|
|
- `skill_metrics`: List in the format of the Skill module output, containing the skill metrics as named arrays.
|
|
|
- `probabilities`: List in the format of the Skill module output, containing the 33rd and 66th percentiles.
|
|
|
- If `significance = TRUE` (default), the statistical significance dots will be displayed in the plot, when available. It defaults to FALSE.
|
|
|
|
|
|
`Visualization()` attempts to generate:
|
|
|
|
|
|
- Plots of all the skill metrics provided in `skill_metrics`.
|
|
|
- A plot of the forecast ensemble mean, if a forecast has been provided.
|
|
|
- A Most Likely Terciles plot, if the `probabilities` include the terciles (percentiles 33 and 66).
|
|
|
|
|
|
The three functions that `Visualization()` calls can also be called independently. In this case, archive configuration file and the the output directory for the plots will have to be provided explicitly as arguments to :
|
|
|
|
|
|
**plot_skill_metrics(recipe, archive, data_cube, skill_metrics, outdir, significance = F)**: Generates, for each metric in skill_metrics, a figure with one plot per time step, and saves each figure to the output directory `outdir` as `<metric>.png`.
|
|
|
|
|
|
`data_cube` is an s2dv_cube containing the appropriate metadata, for example the hcst object that was passed to the Skill module.
|
|
|
|
|
|
**plot_ensemble_mean(recipe, archive, fcst, outdir)**: Computes the forecast ensemble mean and generates a figure with one plot per time step, and saves it to the output directory `outdir` as `forecast_ensemble_mean.png`
|
|
|
|
|
|
**plot_most_likely_terciles(recipe, archive, fcst, percentiles, outdir)**: Computes the forecast tercile probability bins with respect to the terciles provided in 'percentiles', then generates a figure with one plot per time step and saves it to the directory `outdir` as `forecast_most_likely_terciles.png`
|
|
|
## Introduction
|
|
|
|
|
|
This the GitLab repository for the SUNSET (SUbseasoNal to decadal climate forecast post-processIng and asSEmenT suite), a modular tool for subseasonal to seasonal to decadal forecast verification workflows. It is intended to have a modularized structure, where each module is a separate part of the code that performs a specific task, so that parts of the workflow can be skipped or reordered.
|
... | ... | @@ -413,9 +6,9 @@ The datasets, forecast horizon, time period, skill metrics to compute and other |
|
|
|
|
|
After definining the recipe, the user can create a script using the functions available in the tool, which use apply the information in the recipe to do the desired computations. You an example script [in the SUNSET code Snippets](https://earth.bsc.es/gitlab/es/sunset/-/snippets/111).
|
|
|
|
|
|
- Modules currently available: Loading, Calibration, Anomalies, Skill, Indices, Saving, Visualization
|
|
|
- Modules in development: Downscaling, Scorecards
|
|
|
- Future modules: Aggregation, Indicators
|
|
|
- Modules currently available: Loading, Units, Calibration, Downscaling, Anomalies, Skill, Indices, Saving, Visualization, Scorecards
|
|
|
- Modules in development: Indicators, Statistics
|
|
|
- Future modules: Aggregation
|
|
|
|
|
|
This tool is in the early stages of development, so the code and the information in this wiki may be subject to frequent changes and updates. This wiki contains all the information needed to use the available modules.
|
|
|
|
... | ... | @@ -704,9 +297,9 @@ The outputs are saved to a unique folder inside the directory you specified in t |
|
|
|
|
|
The structure of the subdirectories and file names will depend on the selected '`Output_format`' in the recipe.
|
|
|
|
|
|
There are two possible output formats currently: `'S2S4E'` and `'Scorecards'`, described below. To request the inclusion of an additional output format, please open an issue. It is also possible to save the data to a custom directory in a custom format, by making use of the CSTools functions `SaveExp()` and `CST_SaveExp()`.
|
|
|
There are two possible output formats currently: `'S2S4E'` and `'Scorecards'`, described below. To request the inclusion of an additional output format, please open an issue.
|
|
|
|
|
|
#### S2S4E (default)
|
|
|
#### S2S4E format (default)
|
|
|
With the 'S2S4E' output format, the structure of the subdirectory and the files is as follows:
|
|
|
|
|
|
If fcst_year has been requested:
|
... | ... | @@ -730,7 +323,7 @@ The structure of the names of the files is: |
|
|
|
|
|
Where `var` is the name of the variable, `yyyy` is the year, `mm` is the initialization month and `mm` is the initialization day.
|
|
|
|
|
|
#### Scorecards:
|
|
|
#### Scorecards format:
|
|
|
For the 'Scorecards' output format, the structure of the subdirectory and the files is:
|
|
|
|
|
|
- Skill metrics: `<system>/<variable>/scorecards_<system>-<reference>_<variable>-skill_<startyear>_<endyear>_s<mm>.nc`
|
... | ... | @@ -762,6 +355,8 @@ Saving(recipe, data, skill_metrics = skill_metrics) |
|
|
Saving(recipe, data, probabilities = probabilities)
|
|
|
```
|
|
|
|
|
|
It is also possible to save the data to a custom directory in a custom format, by making use of the CSTools functions `SaveExp()` and `CST_SaveExp()`.
|
|
|
|
|
|
## Visualization module
|
|
|
|
|
|
The Visualization module provides a few basic plots to visualize the data loaded and computed using the previous modules. The color palettes and titles of the plots will vary depending on the output format requested in the recipe. The palettes are generated using the clim.colors() function from the [s2dv](https://cran.r-project.org/web/packages/s2dv/index.html) package.
|
... | ... | |