I finished all the unit tests (the three shown in the previous comment and two more). The unit tests verify the structure of the generated provenance graph—including node names, node positions, and the number of nodes—at each step of the workflow. They also confirm that the structure of the generated JSON file is correct.
test_provenance4.R
source("modules/Loading/Loading.R")
source("modules/Units/Units.R")
source("modules/Calibration/Calibration.R")
source("modules/Anomalies/Anomalies.R")
source("modules/Skill/Skill.R")
source("modules/Saving/Saving.R")
source("modules/Visualization/Visualization.R")
source("SUNSET_PROV/R/SunsetProv.Command.R")
library(igraph)
library(rjson)
library(testthat)
#SUNSET loading module execution
recipe_file <- "tests_provenance/recipes/recipe_test_provenance4.yml"
#Loading MODULE
recipe <- prepare_outputs(recipe_file)
data <- Loading(recipe)
#Calibration MODULE
data_cal <- Calibration(recipe, data)
#Anomalies MODULE
data_ano <- Anomalies(recipe, data_cal)
#Compute skill metrics
skill_metrics <- Skill(recipe, data_ano)
#Compute percentiles and probability bins
probabilities <- Probabilities(recipe, data_ano)
#Define provenance graph for Visualization
graph.prov <- skill_metrics$graph.prov
#Visualization
Visualization(recipe, data, graph.prov,
skill_metrics = skill_metrics,
probabilities = probabilities,
significance = TRUE)
#Generate JSON file
graph2json(graph.prov$graph,
"/esarchive/scratch/apuiggro/sunset/tests_provenance/out-logs/json_test4.json")
test_provenance5.R
source("modules/Loading/Loading.R")
source("modules/Calibration/Calibration.R")
source("modules/Skill/Skill.R")
source("modules/Saving/Saving.R")
source("modules/Visualization/Visualization.R")
source("SUNSET_PROV/R/SunsetProv.Command.R")
library(igraph)
library(rjson)
library(testthat)
recipe_file <- "tests_provenance/recipes/recipe_test_provenance5.yml"
#Loading MODULE
recipe <- prepare_outputs(recipe_file)
archive <- read_yaml("conf/archive_decadal.yml")
# Load datasets
data <- Loading(recipe)
# Calibrate datasets
data_cal <- Calibration(recipe, data)
# Compute skill metrics
skill_metrics <- Skill(recipe, data_cal)
probs <- Probabilities(recipe, data_cal)
graph.prov <- skill_metrics$graph.prov
# Plotting
Visualization(recipe = recipe, data = data_cal, graph.prov,
skill_metrics = skill_metrics, probabilities = probs,
significance = T )
#Generate JSON file
graph2json(graph.prov$graph,
"/esarchive/scratch/apuiggro/sunset/tests_provenance/out-logs/json_test5.json")
Finished the set of functions to describe a SUNSET execution workflow for atomic recipes and integrated the functions inside the modules. The functions are almost fully independent of the metaclipR package. Created three tests that can be run to reproduce the provenance results by cloning the dev-test-provenance branch of SUNSET. The aim is to make them unit tests!
test_provenance1.R
:
source("modules/Loading/Loading.R")
source("modules/Units/Units.R")
source("modules/Calibration/Calibration.R")
source("modules/Anomalies/Anomalies.R")
source("modules/Skill/Skill.R")
source("modules/Saving/Saving.R")
source("modules/Visualization/Visualization.R")
source("modules/Indices/Indices.R")
source("modules/Downscaling/Downscaling.R")
library(igraph)
#Recipe
recipe_file <- "recipes/atomic_recipes/recipe_test_provenance1.yml"
#Loading recipe
recipe <- prepare_outputs(recipe_file)
#Loading module
data <- Loading(recipe)
#Units module
data <- Units(recipe, data)
#Calibration
data <- Calibration(recipe, data)
#Anomalies
data <- Anomalies(recipe, data)
#Compute skill metrics
skill_metrics <- Skill(recipe, data)
#Compute percentiles and probability bins
probabilities <- Probabilities(recipe, data)
#Define provenance graph for Visualization
graph.prov <- skill_metrics$graph.prov
#Visualization
Visualization(recipe, data, graph.prov,
skill_metrics = skill_metrics,
probabilities = probabilities,
significance = TRUE)
#Generate JSON file
graph2json(graph.prov$graph,
"/esarchive/scratch/apuiggro/sunset/graph333333.json")
test_provenance2.R:
source("modules/Loading/Loading.R")
source("modules/Units/Units.R")
source("modules/Calibration/Calibration.R")
source("modules/Anomalies/Anomalies.R")
source("modules/Skill/Skill.R")
source("modules/Saving/Saving.R")
source("modules/Visualization/Visualization.R")
source("modules/Indices/Indices.R")
source("modules/Downscaling/Downscaling.R")
library(igraph)
# Read recipe
recipe_file <- "recipes/atomic_recipes/recipe_test_provenance2.yml"
recipe <- prepare_outputs(recipe_file)
#Load datasets
data <- Loading(recipe)
#Compute anomalies
data <- Anomalies(recipe, data)
#Compute El Nino indices
nino_data <- Indices(recipe, data)
#Compute skill metrics
graph.prov <- nino_data$graph.prov
nino_data$graph.prov <- NULL
for (index in nino_data) {
index$graph.prov <- graph.prov
nino_skill_metrics <- Skill(recipe = recipe, data = index, agg = "region")
graph.prov <- nino_skill_metrics$graph.prov
nino_skill_metrics$graph.prov <- NULL
}
#Generate JSON file
graph2json(graph.prov$graph,
"/esarchive/scratch/apuiggro/sunset/graph333333.json")
test_provenance3.R:
source("modules/Loading/Loading.R")
source("modules/Skill/Skill.R")
source("modules/Downscaling/Downscaling.R")
source("modules/Saving/Saving.R")
source("modules/Anomalies/Anomalies.R")
recipe_file <- "recipes//atomic_recipes/recipe_test_provenance3.yml"
recipe <- prepare_outputs(recipe_file, disable_checks = F)
#Load datasets
data <- Loading(recipe)
#Compute anomalies
ano_data <- Anomalies(recipe, data)
#Downscale the data
downscaled_data <- Downscaling(recipe, ano_data)
#Compute skill metrics
skill_metrics <- Skill(recipe, downscaled_data)
#Generate JSON file
graph2json(skill_metrics$graph.prov$graph,
"/esarchive/scratch/apuiggro/sunset/graph333333.json")
I will add a fourth test that includes Socrecards and loads "fcst", also a fifth test including a decadal execution. All the code is in the folder SUNSET_PROV
of the SUNSET repository branch. The test files can be found in the main folder under test_provenance1.R
, test_provenance2.R
and test_provenance3.R
Finished the set of functions to describe a SUNSET execution workflow for atomic recipes, almost fully independent from the metaclipR package. Most provenance information is extracted directly from the recipe or the s2dv_cube on the fly. The igraph object is included inside the s2dv_cube as data$graph.prov
and uploaded every time a module is executed. Functions are located in esarchive/scratch/apuiggro/sunset/SUNSET_PROV
and work modularly as SUNSET functions do. Also created a conf/
file where all information regarding calibration methods, interpolation methods, and metrics is stored. The resulting graphs look quite complex.
If we can finally achieve a vocabulary extension, more specific terms will be added to describe each transformation. Now, I have to document the functions and revise them to ensure their proper functioning.
I managed to develop two more provenance functions (sunset_prov_downscale and sunset_prov_verification), which quite push the METACLIP ontology. If we eventually expand the ontology, we would simply need to update the vocabularies used with the more "specific" ones. The last function to be implemented will be 'sunset_prov_visualization', which is currently in progress.
At present, the functions are hybrids, considering that metaclipR functions are utilized in most functions combined with functions from the igraph package. We could consider developing our own functions to gain specificity to SUNSET and directly extract provenance information from the s2dv_cube (wouldn't take too long). This needs further discussion.
For now, a standard result appears as follows(We can identify the Member realization, the Downscaling step from the Anomalies as a 'cal: withCalibrationMethod', and the 'veri: Verification' node connected to the computed Skill metrics with the vertex 'veri:withForecastRepresentation'):
LINK TO DRIVE DOC: https://docs.google.com/document/d/121MUDlMNIvF0G-R61JzqpouCKXb4SBqI5RtFdqvO5ro/edit?usp=sharing
@nperez and @vagudets feel free to add there any comment or modification you consider!
Done!
Sorry! I know the comment is too long...
Second draft of how we could address the ontology expansion. Considering that most of the added terms align well with the current ontology, I've proposed extending the already defined ontologies cal:
and val
:. Additional individuals may be included in the future. Currently, such an extension could enable us to generate nearly complete provenance for SUNSET. It's important to note that this proposed expansion is still preliminary and requires thorough review.
(All definitions need to be reviewed before presenting them to the METACLIP maintainers.)
GITHUB ISSUE: https://github.com/metaclip/vocabularies/issues/5#issuecomment-2037878749
Each element (either class, individual or object property in this particular case) is defined by means of "Annotation properties". The most used ones are defined as follows:
Also, remark the fact that individuals refer to real worlds objects, or events, and types, or classes, to sets of real world objects. Class expressions or definitions gives the properties that the individuals must fulfill to be members of the class. Also, object properties are used to assert relationships between individuals (or instances).
prov:Entity⇒
cal:CalibrationMethod⇒
cal:ESD⇒
cal:ESD-CSDownscale
*dc:title:"Empirical Statistical Downscaling in R package CSDownscale"
*dc:description:"R package CSDownscale provides a set of statistical downscaling methods for climate predictions, ready to be applied to refine the output of climate predictions"
prov:Entity⇒
prov:Influence⇒
prov:EntityInfluence⇒
prov:Derivation⇒
ds:Step⇒
ds:Transformation⇒
cal:Calibration⇒
cal:Downscaling
*dc:description:“Downscaling refers to the process of refining or disaggregating coarse-scale or large-scale data to a finer spatial or temporal resolution.”
*seeAlso: https://en.wikipedia.org/wiki/Downscaling
prov:Entity⇒
val:VerificationMetrics
*dc:description: "Verification metrics are used to assess the performance and skill of system models by comparing their predictions with observed data"
seeAlso: https://www.cawcr.gov.au/projects/verification/verif_web_page.html
All individuals listed below are instances of cal: ParametricBiasCorrection
cal:CalibrationMethod ⇒
cal:BiasCorrection ⇒
cal: ParamatricBiasCorrection
BiasOnly
dc:description: "This method corrects the bias in the system. It does not address variance discrepancies between the system and observation"
comment: "For more details, see the CSTools documentation for CST_Calibration()"
hasProbCharacter: "deterministic"
hasSeasonality: "monthly"
referenceURL: "https://cran.r-project.org/web/packages/CSTools/CSTools.pdf"
EVMOS
dc:title: "Error Variance Minimization Objective Score"
dc:description:"This method not only corrects the bias but also applies a variance inflation technique to ensure that the corrected system maintains correspondence with the variance of the observation. It aims to adjust both bias and variance."
comment: "For more details, see the CSTools documentation for CST_Calibration()"
hasProbCharacter: "deterministic"
hasSeasonality: "monthly"
referenceURL: "https://cran.r-project.org/web/packages/CSTools/CSTools.pdf"
isDefinedBy: "Van Schaeybroeck and Vannitsem, 2011"
MSE_min
dc:title:"Mean Squared Error (Minimizing)"
dc:description:"This ensemble calibration method aims to correct bias, overall system variance, and ensemble spread. It minimizes a constrained mean-squared error using three parameters. The goal is to adjust the system to match the observed variance and spread."
comment: "For more details, see the CSTools documentation for CST_Calibration()"
hasProbCharacter: "deterministic"
hasSeasonality: "monthly"
referenceURL: "https://cran.r-project.org/web/packages/CSTools/CSTools.pdf"
isDefinedBy: "Doblas-Reyes et al. (2005)"
CRPS
dc:title:"Continuous Ranked Probability Score (Minimizing)"
dc:description:"This ensemble calibration method corrects bias, overall system variance, and ensemble spread. It does so by minimizing the Continuous Ranked Probability Score (CRPS) using four parameters. CRPS is a scoring rule that evaluates the probabilistic systems."
comment: "For more details, see the CSTools documentation for CST_Calibration()"
hasProbCharacter: "deterministic"
hasSeasonality: "monthly"
referenceURL: "https://cran.r-project.org/web/packages/CSTools/CSTools.pdf"
RPC
dc:title: "Rank Probability Score"
dc:description:"This method adjusts the system variance to ensure that the ratio of predictable components (RPC) is equal to one. It aims to balance the system variance in relation to the predictability of the system."
comment: "For more details, see the CSTools documentation for CST_Calibration()"
hasProbCharacter: "deterministic"
hasSeasonality: "monthly"
referenceURL: "https://cran.r-project.org/web/packages/CSTools/CSTools.pdf"
All individuals listed below are instances of cal: ESD-CSDownscale
cal:CalibrationMethod ⇒
cal:ESD ⇒
cal: ESD-CSDownscale
Int
dc:title: "Interpolation"
dc:description:"Regrid of a coarse-scale grid into a finescale grid, or interpolate model data into a point location. Different interpolation methods, based on different mathematical approaches, can be applied: conservative, nearest neighbor, bilinear or bicubic. Does not rely on any data for training."
dc:comment: "For more details, see the CSTools documentation for CSDownscale"
referenceURL:"https://earth.bsc.es/gitlab/es/csdownscale"
hasProbCharacter: "deterministic"
hasSeasonality: "monthly"
isMultiSite: true?
isMultiVariable: false?
IntLR
dc:title: "Interpolation with Linear Regression"
dc:description:"Interpolate model data into a fine-scale grid or point location. Later, a linear-regression with the interpolated values is fitted using high-res observations as predictands, and then applied with model data to correct the interpolated values."
dc:comment: "For more details, see the CSTools documentation for CSDownscale"
referenceURL:"https://earth.bsc.es/gitlab/es/csdownscale"
hasProbCharacter: "deterministic"
hasSeasonality: "monthly"
isMultiSite: true?
isMultiVariable: false?
IntBC
dc:title: "Interpolation with Bias Correction"
dc:description:"Interpolate model data into a fine-scale grid or point location. Later, a bias adjustment of the interpolated values is performed. Bias adjustment techniques include simple bias correction, calibration or quantile mapping"
dc:comment: "For more details, see the CSTools documentation for CSDownscale"
referenceURL:"https://earth.bsc.es/gitlab/es/csdownscale"
hasProbCharacter: "deterministic"
hasSeasonality: "monthly"
isMultiSite: true?
isMultiVariable: false?
LogReg
dc:title: "Logistic Regression"
dc:description:"Relate ensemble mean anomalies of the large-scale forecasts directly to probabilities of observing above normal/normal/below normal conditions at the local scale using a sigmoid function. It does not produce an ensemble of forecasts but rather their associated probabilities. It is a statistical method with few parameters to train, and only benefits from local information, but it has shown good performance."
dc:comment: "For more details, see the CSTools documentation for CSDownscale"
referenceURL:"https://earth.bsc.es/gitlab/es/csdownscale"
hasProbCharacter: "deterministic"
hasSeasonality: "monthly"
isMultiSite: true?
isMultiVariable: false?
Analogs
dc:title: "Analogs"
dc:description:"Analog interpolation, or Analog Ensemble (AnEn) interpolation, utilizes historical situations resembling current conditions to estimate values at unsampled locations. It selects a subset of similar historical events (analog situations), assigns weights based on similarity, and computes a weighted average of analog values to interpolate values at the target location, offering an alternative approach to traditional interpolation methods. This method captures spatial variability and provides uncertainty estimates, particularly beneficial in areas with complex terrain or sparse observational data."
dc:comment: "For more details, see the CSTools documentation for CSDownscale"
referenceURL: "https://earth.bsc.es/gitlab/es/csdownscale"
hasProbCharacter: "deterministic"
hasSeasonality: "monthly"
isMultiSite: true?
isMultiVariable: false?
prov:Entity⇒
ver: VerificationMetrics
EnsCorr
dc:title: "Ensemble Mean Correlation"
dc:description: "Calculate correlation between forecasts and observations for an ensemble forecast, including an adjustment for finite ensemble sizes."
dc:comment: "if \_specs
is added at the end of the metric (e.g. FRPSS_specs), the values will be computed using SpecsVerification."
referenceURL: https://cran.r-project.org/web/packages/SpecsVerification/SpecsVerification.pdf
seeAlso: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
corr_individual_members
dc:title:"Correlation for each individual member of the ensemble"
dc:description:???
ReferenceURL: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
RPS
dc:title:"Ranked Probability Score"
dc:description: "The Ranked Probability Score (RPS; Wilks, 2011) is defined as the sum of the squared differences between the cumulative forecast probabilities (computed from the ensemble members) and the observations (defined as 0 did not happen and 100 of multi-categorical probabilistic forecasts. The RPS ranges between 0 (perfect forecast) and n-1 (worst possible forecast), where n is the number of categories. In the case of a forecast divided into two categories (the lowest number of categories that a probabilistic forecast can have), the RPS corresponds to the Brier Score (BS; Wilks, 2011), therefore ranging between 0 and 1. The function first calculates the probabilities for forecasts and observations, then use them to calculate RPS. Or, the probabilities of exp and obs can be provided directly to compute the score. If there is more than one dataset, RPS will be computed for each pair of exp and obs data. The fraction of acceptable NAs can be adjusted."
ReferenceURL: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
RPSS
dc:title:"Ranked Probability Skill Score"
dc:description: "The Ranked Probability Skill Score (RPSS; Wilks, 2011) is the skill score based on the Ranked Probability Score (RPS; Wilks, 2011). It can be used to assess whether a forecast presents an improvement or worsening with respect to a reference forecast. The RPSS ranges between minus infinite and 1. If the RPSS is positive, it indicates that the forecast has higher skill than the reference forecast, while a negative value means that it has a lower skill. Examples of reference forecasts are the climatological forecast (same probabilities for all categories for all time steps), persistence, a previous model version, and another model. It is computed as RPSS = 1 - RPS_exp / RPS_ref. The statistical significance is obtained based on a Random Walk test at the specified confidence level (DelSole and Tippett, 2016). The function accepts either the ensemble members or the probabilities of each data as inputs. If there is more than one dataset, RPSS will be computed for each pair of exp and obs data. The NA ratio of data will be examined before the calculation. If the ratio is higher than the threshold (assigned by parameter na.rm), NA will be returned directly. NAs are counted by per-pair method, which means that only the time steps that all the datasets have values count as non-NA values. "
ReferenceURL: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
FRPS
dc:title: "Fair Ranked Probability Score"
dc:description: "Calculate the difference (mean score of the reference forecast) minus (mean score of the forecast). Uncertainty is assessed by the Diebold-Mariano test for equality of predictive accuracy."
dc:comment: "if \_specs
is added at the end of the metric (e.g. FRPSS_specs), the values will be computed using SpecsVerification."
referenceURL: https://cran.r-project.org/web/packages/SpecsVerification/SpecsVerification.pdf
seeAlso: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
FRPSS
dc:title: "Fair Ranked Probability Skill Score"
dc:description: "Calculate the difference (mean skill score of the reference forecast) minus (mean skill score of the forecast). Uncertainty is assessed by the Diebold-Mariano test for equality of predictive accuracy. "
dc:comment: "if \_specs
is added at the end of the metric (e.g. FRPSS_specs), the values will be computed using SpecsVerification."
referenceURL: https://cran.r-project.org/web/packages/SpecsVerification/SpecsVerification.pdf
seeAlso: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
CRPS
dc:title: "Continuous Ranked Probability Score"
dc:description: "The Continuous Ranked Probability Score (CRPS; Wilks, 2011) is the continuous version of the Ranked Probability Score (RPS; Wilks, 2011). It is a skill metric to evaluate the full distribution of probabilistic forecasts. It has a negative orientation (i.e., the higher-quality forecast the smaller CRPS) and it rewards the forecast that has probability concentration around the observed value. In case of a deterministic forecast, the CRPS is reduced to the mean absolute error. It has the same units as the data. The function is based on enscrps_cpp from SpecsVerification. If there is more than one dataset, CRPS will be computed for each pair of exp and obs data. "
seeAlso: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
CRPSS
dc:title: "Continuous Ranked Probability Skill Score"
dc:description: "The Continuous Ranked Probability Skill Score (CRPSS; Wilks, 2011) is the skill score based on the Continuous Ranked Probability Score (CRPS; Wilks, 2011). It can be used to assess whether a forecast presents an improvement or worsening with respect to a reference forecast. The CRPSS ranges between minus infinite and 1. If the CRPSS is positive, it indicates that the forecast has higher skill than the reference forecast, while a negative value means that it has a lower skill. Examples of reference forecasts are the climatological forecast, persistence, a previous model version, or another model. It is computed as ’CRPSS = 1 - CRPS_exp / CRPS_ref. The statistical significance is obtained based on a Random Walk test at the specified confidence level (DelSole and Tippett, 2016)."
ReferenceURL: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
BSS10
dc:title: "Brier Skill Score (lower extreme)"
dc:description: "Compute the lower extreme Brier score (BS) and the components of its standard decompostion with the two withinbin components described in Stephenson et al., (2008). It also returns the bias-corrected decomposition of the BS (Ferro and Fricker, 2012). It has the climatology as the reference forecast."
dc:comment: "if \_specs
is added at the end of the metric (e.g. FRPSS_specs), the values will be computed using SpecsVerification."
referenceURL: https://cran.r-project.org/web/packages/SpecsVerification/SpecsVerification.pdf
seeAlso: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
BSS90
dc:title: "Brier Skill Score (upper extreme)"
dc:description: "Compute the upper extreme Brier score (BS) and the components of its standard decompostion with the two withinbin components described in Stephenson et al., (2008). It also returns the bias-corrected decomposition of the BS (Ferro and Fricker, 2012). It has the climatology as the reference forecast."
dc:comment: "if \_specs
is added at the end of the metric (e.g. FRPSS_specs), the values will be computed using SpecsVerification."
referenceURL: https://cran.r-project.org/web/packages/SpecsVerification/SpecsVerification.pdf
seeAlso: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
MeanBias
dc:title: "Mean bias of the ensemble"
dc:description: "The Mean Bias or Mean Error (Wilks, 2011) is defined as the mean difference between the ensemble mean forecast and the observations. It is a deterministic metric. Positive values indicate that the forecasts are on average too high and negative values indicate that the forecasts are on average too low. It also allows to compute the Absolute Mean Bias or bias without temporal mean. If there is more than one dataset, the result will be computed for each pair of exp and obs data"
ReferenceURL: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
MeanBiasSS
dc:title: "Mean Bias Skill Score"
dc:description: "The Absolute Mean Bias Skill Score is based on the Absolute Mean Error (Wilks, 2011) between the ensemble mean forecast and the observations. It measures the accuracy of the forecast in comparison with a reference forecast to assess whether the forecast presents an improvement or a worsening with respect to that reference. The Mean Bias Skill Score ranges between minus infinite and 1. Positive values indicate that the forecast has higher skill than the reference forecast, while negative values indicate that it has a lower skill. Examples of reference forecasts are the climatological forecast (average of the observations), a previous model version, or another model. It is computed as AbsBiasSS = 1 - AbsBias_exp / AbsBias_ref. The statistical significance is obtained based on a Random Walk test at the confidence level specified (DelSole and Tippett, 2016). If there is more than one dataset, the result will be computed for each pair of exp and obs data."
ReferenceURL: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
EnsSprErr
dc:title: "Ensemble Spread-to-Error Ratio"
dc:description: ??
ReferenceURL: ??
MSSS
dc:title: "Mean Square Skill Score"
dc:description: "Compute the mean square error skill score (MSSS) between an array of forecast ’exp’ and an array of observation ’obs’. The two arrays should have the same dimensions except along ’dat_dim’ and ’memb_dim’. The MSSSs are computed along ’time_dim’, the dimension which corresponds to the start date dimension. MSSS computes the mean square error skill score of each exp in 1:nexp against each obs in 1:nobs which gives nexp * nobs MSSS for each grid point of the array. The p-value and significance test are optionally provided by an one-sided Fisher test or Random Walk test."
ReferenceURL:https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
MSE
dc:title: "Mean Square Error"
dc:description: "Compute the mean square error for an array of forecasts and an array of observations. The MSEs are computed along time_dim, the dimension which corresponds to the start date dimension. If comp_dim is given, the MSEs are computed only if obs along the comp_dim dimension are complete between limits[1] and limits[2], i.e. there are no NAs between limits[1] and limits[2]. This option can be activated if the user wants to account only for the forecasts for which the corresponding observations are available at all leadtimes. The confidence interval is computed by the chi2 distribution."
ReferenceURL: https://cran.r-project.org/web/packages/s2dv/s2dv.pdf
(...COULD ADD ALL THE DataProviders/ModellingCenters and a Project such as SUNSET...)
ds:hadDerivation⇒
ds:hadTransformation⇒
cal:hadCalibration ⇒
cal: hadDownscaling
dc:description: "Object property linking a ds:Step with a cal:Downscaling step.
cal:withCalibrationMethod⇒
cal: withDownscalingMethod
dc:description: “Object property linking a cal:Downscaling with the description of the cal:ESD-CSDownscale applied”
val:withVerificationMetric
dc:description:“Object property linking a ver:Verification step with the description of the ver: VerificationMetric applied”
Okay, thank you for your insight @vagudets ! I'll work on addressing all the issues and improving the definitions. Additionally, I'll strive to clarify the Data and Annotation properties as much as possible to ensure they're easily understandable for the CST team.
First draft of how we could adress the ontology expasion. Considering the fact that most of the added terms are fit well in the current ontology, I've considered the extension inside the already defined ontologies cal:
and val:
. More Individuals could be added in the future. For now, an extension of this type could allow us to generate almost full provenance for SUNSET.
All definitions need to be reviewed before presenting them to the METACLIP maintainers.
GITHUB ISSUE: https://github.com/metaclip/vocabularies/issues/5#issuecomment-2037878749
CLASSES
cal:ESD-CSDownscale
prov:Entity⇒
cal:CalibrationMethod⇒
cal:ESD⇒
cal:ESD-CSDownscale
*dc:description "R package CSDownscale provides a set of statistical downscaling methods for climate predictions, ready to be applied to refine the output of climate predictions"
cal:Downscaling
prov:Entity⇒
prov:Influence⇒
prov:EntityInfluence⇒
prov:Derivation⇒
ds:Step⇒
ds:Transformation⇒
ds:Downscaling
*dc:description “Downscaling refers to the process of refining or disaggregating coarse-scale or large-scale data to a finer spatial or temporal resolution.”
*seeAlso https://en.wikipedia.org/wiki/Downscaling
INDIVIDUALS
Calibration Methods used in Calibration module
cal:CalibrationMethod ⇒
cal:BiasCorrection ⇒
cal: ParamatricBiasCorrection
BiasOnly: "This method corrects only the bias in the system. It does not address variance discrepancies between the system and observation"
EVMOS:"This method not only corrects the bias but also applies a variance inflation technique to ensure that the corrected system maintains correspondence with the variance of the observation. It aims to adjust both bias and variance."
MSE: "This ensemble calibration method aims to correct bias, overall system variance, and ensemble spread. It minimizes a constrained mean-squared error using three parameters. The goal is to adjust the system to match the observed variance and spread."
CRPS: "This ensemble calibration method also corrects bias, overall system variance, and ensemble spread. However, it does so by minimizing the Continuous Ranked Probability Score (CRPS) using four parameters. CRPS is a scoring rule that evaluates the probabilistic systems."
RPC:"This method adjusts the system variance to ensure that the ratio of predictable components (RPC) is equal to one. It aims to balance the system variance in relation to the predictability of the system."
*hasProbCharacter:
deterministic/stocastic
*hasSeasonality:
monthly
*isMultiSite:
false?
*isMultiVariable:
false?
Downscaling methods used in Downscaling module
cal:CalibrationMethod ⇒
cal:ESD ⇒
cal: ESD-CSDownscale
Interpolation:"Regrid of a coarse-scale grid into a finescale grid, or interpolate model data into a point location. Different interpolation methods, based on different mathematical approaches, can be applied: conservative, nearest neighbour, bilinear or bicubic. Does not rely on any data for training."
InterpolationLinearRegression: "Interpolate model data into a fine-scale grid or point location. Later, a linear-regression with the interpolated values is fitted using high-res observations as predictands, and then applied with model data to correct the interpolated values."
InterpolationBias: "Interpolate model data into a fine-scale grid or point location. Later, a bias adjustment of the interpolated values is performed. Bias adjustment techniques include simple bias correction, calibration or quantile mapping"
LogisticRegression: "Relate ensemble mean anomalies of the large-scale forecasts directly to probabilities of observing above normal/normal/below normal conditions at the local scale using a sigmoid function. It does not produce an ensemble of forecasts but rather their associated probabilities. It is a statistical method with few parameters to train, and only benefits from local information, but it has shown good performance."
Analogs:"Analog interpolation, or Analog Ensemble (AnEn) interpolation, utilizes historical situations resembling current conditions to estimate values at unsampled locations. It selects a subset of similar historical events (analog situations), assigns weights based on similarity, and computes a weighted average of analog values to interpolate values at the target location, offering an alternative approach to traditional interpolation methods. This method captures spatial variability and provides uncertainty estimates, particularly beneficial in areas with complex terrain or sparse observational data."
*hasProbCharacter:
deterministic/stocastic
*hasSeasonality:
monthly
*isMultiSite:
false?
*isMultiVariable:
false?
Verification metrics used in Skill module
skos: Concept⇒
ver:Skill score
EnsCorr: Ensemble Mean Correlation.
corr_individual_members: Correlation for each individual member of the ensemble.
RPS: Ranked Probability Score.
RPSS: Ranked Probability Skill Score.
FRPS: Fair Ranked Probability Score.
FRPSS: Fair Ranked Probability Skill Score.
CRPS: Continuous Ranked Probability Score.
CRPSS: Continuous Ranked Probability Skill Score.
BSS10: Brier Skill Score (lower extreme).
BSS90: Brier Skill Score (upper extreme).
Mean_Bias: Mean bias of the ensemble.
Mean_Bias_SS: Mean Bias Skill Score.
EnsSprErr: Ensemble Spread-to-Error Ratio.
MSSS: Mean Square Skill Score
MSE: Mean Square Error
(...COULD ADD ALL THE DataProviders/ModellingCenters and a Project such as SUNSET…)
OBJECT PROPERTIES
hadDownscaling
ds:hadDerivation⇒ ds:hadTransformation⇒ cal:hadCalibration ⇒ cal: hadDownscaling
dc:description "Object property linking a :Downscaling step with the description of the :ESD-CSDownscale method applied"
withDownscalingMethod
cal:withCalibrationMethod⇒ cal: withDownscalingMethod
dc:description “Object property linking a :Calibration step with the description of the :VerificationMetrics applied”
withVerificationMetric
val:withVerificationMetric
dc:description “Object property linking a :Verification step with the description
Since we have to create new functions because metaclipR is lacking functions for some of our processes, my suggestion is that the new functions should use the s2dv_cube data structure directly. If I'm not wrong, you already had some working example for this, right?
Hi @vagudets, that's a possibility. We could develop our own functions based on metaclipR's but fully focused on the s2dv_cube object and adapted to the METACLIP ontology. Also, yes, I have an example in which I developed a "Dataset provenance" function.
If (and only if) it is easy and quick, then we can consider developing our own functions to replace the existing metaclipR ones.
Considering the fact that we will have to develop our own functions for the Skill, Downscaling, and Scorecards modules— including an extension of the current ontology— we could do the same with the other modules. For sure, it would take more time to develop, but we could obtain more specificity and extract provenance directly from the s2dv_cube. We would only need to use the igraph package then.
REVIEW OF DEVELOPED PROVENANCE FUNCTIONS
"sunset_prov_dataset":
METACLIP |
---|
nearest |
bilinear |
bicubic |
IDW |
spline |
ConservativeRemapping |
SUNSET |
---|
conservative |
bilinear |
bicubic |
distance-weighted |
con2 |
laf |
nn |
"sunset_prov_calibration":
METACLIP |
---|
cal:BiasCorrection |
cal:NonParametricBiasCorrection |
cal:LinearScaling |
cal:NPQuantileMapping |
cal:PQuantileMapping |
cal:ParametricBiasCorrection |
cal:VarianceInflation |
cal:ProbabilityScaling |
cal:EnsembleRecalibration |
SUNSET |
---|
raw |
qmap (Quantile Mapping, for daily data) |
bias |
evmos(Error Variance Minimization Objective Score) |
mse_min (Mean Squared Error, Minimizing) |
crps_min (Continuous Ranked Probability Score) |
rpc_based (Rank Probability Score) |
-What to do with the ds:TrainingGraph class? metaclipR.BiasCorrection() function requires that input, we could either modify the function and make it optional or correctly define a dataset that's a Training dataset.
"sunset_prov_anomalies": -Review "metaclipR.Anomalies" inputs. Same with "metaclipR.Climatology()".
"sunset_prov_indices":
OTHER CONSIDERATIONS
I made some adjustments to both "sunset_prov_dataset.R" and "sunset_prov_calibration.R" to ensure proper extraction of provenance metadata, particularly in cases where a forecast was loaded. Additionally, I introduced a new function called "sunset_prov_anomalies." However, I'm not entirely sure about how the Anomalies module works; I plan to discuss this further with Victoria tomorrow. It's evident that all the developed provenance functions are still in a preliminary stage and require adjustments, primarily due to the necessity of incorporating new vocabularies and ensuring compatibility with any recipe setup.
The provenance graph is starting to look quite impressive, and the generated JSON file is quite lengthy. I need to figure out how to obtain a more "organized" graph. I had the opportunity to test the code with various datasets and recipe commands successfully. The latest test involved loading the Meteo-France-System7 dataset and computing calibration and anomalies. The graph looks like this:
Thanks!
Keep in mind that there is a third option to interpolate all of them to a common grid, which would have to be added to all of the datasets.
I know, I just wasn't sure where to find those grids. Once I can load them, it won't be difficult to implement that option with the metaclipR.SpatialExtent() function.
Where are you keeping the modified functions?
I'm keeping it in "/esarchive/scratch/apuiggro/sunset". At the moment, I've just encountered problems with this one. Just wanted to test if the modifications could actually make it work. Also, not much had to be changed.
I know I have to be a little more organized because I'm starting to have a lot of code. As soon as I can, I'll try to organize everything in the cloned git repository.
I've just pushed into the "dev-test-branch" all the new modifications if you want to have a look. Still very preliminary, just messing around and looking at how we could address the implementation.
Albert
Could advance and include a common command call (as there is acutally only one function call "Loading(recipe)" that computes datasubset and interpolation. Still thinking on how to adress it) and interpolation method based on the recipe input (either to the obs if "to_system", to hcst and fcst if "to_reference"...) Had to modify some aspects of the metaclipR.Interpolation() function due to incompatibility with the package detection... Also, some interploation/regridding methods not established in the ontology...Here are the results:
Would like to try the function with different recipes. Not quite sure which ones can I use, for now I'm focusing on seasonal datasets and the atomic recipe "recipe_seasonal_provenance.yml".
As mentioned in the previous comment, I've developed a simple set of functions to generate dataset/datasubset provenance using information from the recipe, the loaded data (s2dv_cube), and the configuration files archive.yml (for seasonal) and archive_decadal.yml (for decadal). I still want to review the code and add more features. For now, I'm able to obtain the following image in the interpreter when executing this code:
source("modules/Loading/Loading.R")
source("modules/Units/Units.R")
source("tools/s2dv_cube2Climate4R.R")
source("sunset_prov_dataset.R")
source("combination_graph.R")
library(yaml)
library(metaclipR)
#SUNSET loading module execution
recipe_file <- "recipes/atomic_recipes/recipe_seasonal_provenance.yml"
recipe <- prepare_outputs(recipe_file)
data <- Loading(recipe)
#Dataset and DatasetSubset provenance generation
graph<-sunset_prov_dataset(recipe)
#Conversion to JSON file to run the interpreter!
graph2json(graph$graph, "/esarchive/scratch/apuiggro/sunset/graph333.json")
In this particular case, the graph provides provenance information for an observational dataset and a seasonal hindcast
Okay! I'll take a look at it. Additionally, I've developed an example to generate simple data provenance for a loaded hcs and obs using metaclipRDataset() and metaclipRDatasetSubset(). The code is located in the dev-test-provenance branch. I'll further expand it to include calibration and regridding options.
Also, thank you very much @eduzenli for your help!! The function is incredibly useful for us and came at the perfect time, we couldn't have asked for more.