User review: BrierScore and UltimateBrier
Hi @jramon
Thanks for reviewing BrierScore() and UltimateBrier() for us. The functions are in the branch develop-BrierScore. Here are some recommended steps to test them.
- Read the documentation (i.e., the header of the function) to check if there is anything unclear or wrong.
- Find suitable data to test the functions. Note that s2dverification::BrierScore can only work with vectors, but others work with multi-dimensional arrays.
- Some packages may need to be loaded manually, e.g.,
library(multiApply)
. Besides, some other functions in s2dv may be used in the testing functions too, solibrary(s2dv)
can be added to the script. - One way to test the s2dverification and s2dv functions together without confusion is to assign them with different names first. For example,
library(s2dv)
library(multiApply)
BS_old <- s2dverification::BrierScore
source('<path_to_new_function>/BrierScore.R')
BS_new <- BrierScore
res_old <- BS_old(...)
res_new <- BS_new(...)
- Check if the two results are consistent. The dimension order may change after the calculation, and you can use
s2dv::Reorder
to adjust the order back. - Besides the data itself, is there anything to be improved? E.g., the input parameters, the output format, etc.
I tried to keep everything the same as the old functions, but I have some queries and have detected some problems:
[BrierScore]
- There are two functions in s2dverification::BrierScore. BrierScore() works with vector inputs, and .BrierScore() is designed for VeriApply(). s2dv::BrierScore combines them together.
- Note that the inputs of s2dverification::BrierScore are
obs
firstpred
second, but all other functions including UltimateBrier() areexp
firstobs
second. - I removed the last three output items
$pred
,$obs
, and$bins
. The first two were removed because I thought it was redundant to replicate the inputs as outputs. As for$bins
, I couldn't find a way to structure it in the new function which holds arrays instead of vectors. Please let me know if you think keeping these outputs is necessary. - s2dverification:::.BrierScore has
fkbar[fkbar == Inf] <- 0
andokbar[is.nan(okbar)] <- 0
(at line 189 and 190), but s2dveirifcation::BrierScore doesn't have these operations. I don't know if they're necessary. For now, s2dv::BrierScore doesn't include these two lines. - The documentation is not very explanatory for me (e.g., the output terms only have a formula but no explanation), but perhaps it's just because I'm not familiar with Brier score. Do you think it should be improved?
[UltimateBrier]
- The parameter 'thr' doesn't include 0 and 1 (while BrierScore does). Is it error-prone? I don't know which is more common usage.
- 'thr' is used in 'FairEnsembleBSS' and 'FairEnsembleBS' (line 231 and 239). But if quantile = F and thr is real value, does it work? It seems weird to me because
c(thr, 1)
implies thatthr
should be between [0, 1]. - I'm not sure about
thr
usage. It is used in the probabilistic calculation line 214, but should it be used in the following Brier score calculation? For example, line 244, 246, and 248 use BrierScore(), but thethresholds
argument is default, notthr
assigned in UltimateBrier(). For now, I keep s2dv::UltimateBrier same as before. - I think the calculation of FairEnsembleBS is wrong. line 318 calculates the decomposition of FairEnsembleBS but it doesn't have decomposition.
Sorry that these two functions are quite messy and have many small features inside. It is welcomed if you have a better idea to reconstruct them thoroughly. And please let me know if anything is not clear to you. Thank you so much for your help!
Cheers,
An-Chi