Seasonal case testing: Saving module
Hi @cchou, @allabres and @nmilders
First of all, we have a wiki! The user documentation will be written here.
@aho and I have been working on the Saving module and on various improvements for the other modules and just merged all the changes to the master branch. I would like to ask you to go over the new changes, test this module, and let us know if you notice any bugs, and whether the file structure works for you or if there's anything you would like to modify. You can give feedback in the comments here. For more information, please refer to the Saving module section of the wiki.
In the snippets you will find an updated example script.
Below is a list of the main changes you will notice, some of which we have already discussed. Please open an issue if you have trouble with any of them.
Recipe:
- Changes in the Time section: leadtimemin/max to ftime_min/max, and clearer terminology for the start date:
Time:
sdate: '1101' # Mandatory, int: Start date, 'mmdd'
fcst_year: '2020' # Optional, int: Forecast initialization year 'YYYY'
hcst_start: '1993' # Mandatory, int: Hindcast initialization start year 'YYYY'
hcst_end: '2016' # Mandatory, int: Hindcast initialization end year 'YYYY'
ftime_min: 1 # Mandatory, int: First forecast time step in months. Starts at “1”.
ftime_max: 2 # Mandatory, int: Last forecast time step in months. Starts at “1”.
- New option for
Calibration:method
: 'raw' - no calibration will be performed and the raw hindcast and forecast will be returned. - New metrics available: CRPS and CRPSS.
-
Skill:prob
added to request probability category/quantile thresholds. - New parameters:
ncores
andremove_NAs
.
Workflow:
Calibration:
method: mse_min # Mandatory, str: Calibration method.
Skill:
metric: RPSS CRPSS # Mandatory, str: List of skill metrics.
Probabilities:
percentiles: [[1/3, 2/3], [1/10, 9/10], [1/4, 2/4, 3/4]] # Optional: Thresholds
# for quantiles and probability categories. # Each set of thresholds should be
# enclosed within brackets.
Indicators:
index: no # This feature is not implemented yet
ncores: 4 # Optional, int: number of cores to be used in parallel computation.
# If left empty, defaults to 1.
remove_NAs: TRUE # Optional, bool: Whether to remove NAs.
# If left empty, defaults to FALSE.
Here is a link to a complete example for the new recipe.
Main script:
- compute_skill_metrics() now returns a list of named arrays, and not a list of lists. This means that previously,
skill_metrics$rpss[[1]]
was needed to access the RPSS array, now onlyskill_metrics$rpss
is needed. You might need to change your scripts accordingly. - Extra singleton dimensions have been dropped for the metrics. You might also need to change this in your scripts. You can compare the new example script to the old one to see the differences.
- New function
compute_probabilities()
[Skill module]: Computes the requested percentiles and probability bins. You can request the thresholds in the recipe; this is explained in more detail in the Skill module section of the wiki.
This was a lot of information, so please take your time, and let me know if you have any questions! We also plan on holding a user meeting soon.
Cheers,
Victòria