s2dverification issueshttps://earth.bsc.es/gitlab/es/s2dverification/-/issues2019-09-04T11:43:43+02:00https://earth.bsc.es/gitlab/es/s2dverification/-/issues/183Possibility to change easily a plots2019-09-04T11:43:43+02:00Martin MenegozPossibility to change easily a plotsHi,
Very often, we have to make beautiful plots for paper/reports, and often, we need for that to make slight modifications of the s2dverification scripts, to include some specificities to our plots.
In the past, I was copying locally Pl...Hi,
Very often, we have to make beautiful plots for paper/reports, and often, we need for that to make slight modifications of the s2dverification scripts, to include some specificities to our plots.
In the past, I was copying locally PlotAno.R for example, to modify manually one line thickness or color. It was a quick way to do that. And I was not creating a new branch, because I have the impression that my modifications were really small and particularly specific to my application. I did want neither to include too many arguments in the PlotAno.R function, otherwise, we will have soon too many arguments for this function.
Now, we cannot do that any more because of some new functionalities in PlotAno.R and in probably in other functions that prevent the possibility to run them alone after including some small modifications, for example this function in PlotAno.R:
if (!is.null(fileout)) {
deviceInfo <- .SelectDevice(fileout)
saveToFile <- deviceInfo$fun
fileout <- deviceInfo$files
}
@nmanubens, what do you recommend us when we want to, include some small changes in functions like PlotAno.R?
For example, today, I would like just to increase the thickness of the black curve corresponding to the observations in PlotAno.R.
All the best,
Martin
PS: @cprodhomme may be interested in your answerNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/140High performance and scores2019-11-11T18:23:13+01:00Nicolau Manubens GilHigh performance and scoresCurrently, computation of scores or other diagnostics in s2dverification is slow.
It is needed to optimize the functions in terms of:
a) Computing time, for example by using base R vectorised functions or by calling C functions
...Currently, computation of scores or other diagnostics in s2dverification is slow.
It is needed to optimize the functions in terms of:
a) Computing time, for example by using base R vectorised functions or by calling C functions
b) Memory usage, for example by avoiding unneeded copies of arrays in the score functions.
c) Parallelism, particularly making them able to run on clusters with thousands of cores.
@ncortesi already programmed a set of scripts that allow to run score functions by chunks on MareNostrum (or SMP, with less than 128 tasks) and analysed the performance, giving a solution to c). More information can be found at https://earth.bsc.es/wiki/lib/exe/fetch.php?media=library:internal:20160615_running_diagnostics_on_marenostrum.pdf . This solution is, however, not ready to be used without some difficult configuration steps. @nmanubens is in contact with him to solve this issue. Additionally, in the long term, this solution should ideally be ported to R and make it available through parameters in the typical score functions or by evolving the `veriApply()` function in the package easyVerification.
@nmanubens and @jginer are working on a compatibility break in s2dverification in which the data model used throughout will be revised and improved taking into account existing libraries that, for example, offer abstractions of distributed arrays in a way that, a simple sum of two arrays, is launched transparently and automatically in the multiple nodes. Furthermore, this model has to be agreed and common with the one in downscaleR (discussion ongoing with JoaquĆn from UNICAN) and it should also end up being, ideally, the data model used in QA4Seas.
The Data & Diagnostics team is also investigating existing paradigms and techniques to deal with this kind of problems.Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/116Area average option in Load 2015-11-30T18:31:44+01:00Eleftheria ExarchouArea average option in Load Hi,
Since I was puzzled that the s2dverification needs regular grid, even with the "areave" optiom, I made some small test with sst data from nemo and its true that when there is no cell_area in the file, we have the following situati...Hi,
Since I was puzzled that the s2dverification needs regular grid, even with the "areave" optiom, I made some small test with sst data from nemo and its true that when there is no cell_area in the file, we have the following situation
```
eexarcho@moore:/esnas/exp/ecearth/a02p/19930501/fc0/outputs$ cdo fldmean sosstsst.nc sosstsst_fldmean.nc
cdo fldmean (Warning): Using constant grid cell area weights for variable sosstsst!
cdo fldmean: Processed 422816 values from 1 variable over 4 timesteps ( 0.03s )
```
But: if we attach the cell_area with
```
ncks -A -v cell_area area_tmask_nemo.N3.2_O1L42.nc sosstsst.nc
ncatted -a cell_measures,sosstsst,c,c,area: cell_area sosstsst.nc
```
then
```
eexarcho@moore:/esnas/exp/ecearth/a02p/19930501/fc0/outputs$ cdo fldmean sosstsst.nc sosstsst_fldmean_cellarea.nc
cdo fldmean: Processed 422816 values from 1 variable over 4 timesteps ( 0.04s )
```
And apparently the 2 files are different:
```
eexarcho@moore:/esnas/exp/ecearth/a02p/19930501/fc0/outputs$ cdo info sosstsst_fldmean.nc
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter ID
1 : 1993-05-16 11:30:00 0 1 0 : 8.4668 : -1
2 : 1993-06-15 23:00:00 0 1 0 : 8.4538 : -1
3 : 1993-07-16 11:00:00 0 1 0 : 8.4333 : -1
4 : 1993-08-16 11:00:00 0 1 0 : 8.4114 : -1
eexarcho@moore:/esnas/exp/ecearth/a02p/19930501/fc0/outputs$ cdo info sosstsst_fldmean_cellarea.nc
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter ID
1 : 1993-05-16 11:30:00 0 1 0 : 18.090 : -1
2 : 1993-06-15 23:00:00 0 1 0 : 18.062 : -1
3 : 1993-07-16 11:00:00 0 1 0 : 18.006 : -1
4 : 1993-08-16 11:00:00 0 1 0 : 17.961 : -1
```
So, should Load be modified so that when "areave" is chosen, it can accept irregular grid?
Cheers
Eleftheria
People: @nmanubens @vguemas @obellprat @mmenegoz @fmassonnet @nevensf https://earth.bsc.es/gitlab/es/s2dverification/-/issues/86ncdf vs. ncdf4 or both?2015-08-19T16:58:11+02:00Nicolau Manubens Gilncdf vs. ncdf4 or both?As seen in https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CC4QFjACahUKEwj0mrmW_LTHAhULvBQKHarlBK4&url=https%3A%2F%2Fpublicwiki.deltares.nl%2Fdownload%2Fattachments%2F42401998%2FFedorBaart_ncdf4.pdf%3Fversion%3D1%26m...As seen in https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CC4QFjACahUKEwj0mrmW_LTHAhULvBQKHarlBK4&url=https%3A%2F%2Fpublicwiki.deltares.nl%2Fdownload%2Fattachments%2F42401998%2FFedorBaart_ncdf4.pdf%3Fversion%3D1%26modificationDate%3D1286292950000&ei=wF7UVfS6JIv4UqrLk_AK&usg=AFQjCNFChZotf4Z5RgzP_xINhyg1kg6vYQ&sig2=SikGlu6X7pTCIkVh5FuzZA&bvm=bv.99804247,d.d24
ncdf4 breaks windows capabilities whereas ncdf doesn't. Moreover, ncdf is able to load OPeNDAP files too. Should we consider moving back to ncdf or adding ncdf alongside ncdf4 so as not to lose windows compatibility?Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/63Load(): Using masks is sometimes too complex.2015-06-02T01:37:49+02:00Nicolau Manubens GilLoad(): Using masks is sometimes too complex.@cprodhomme , @vtorralba , @mmenegoz , @lpcaron , @lbatte , @obellprat , @nevensf , @eexarchou , @fmassonnet , @aldeppenmeier , @dmacias , @ngonzalez , @dvolpi , @vguemas , @pabretonniere
This issue has been opened as thread to discu...@cprodhomme , @vtorralba , @mmenegoz , @lpcaron , @lbatte , @obellprat , @nevensf , @eexarchou , @fmassonnet , @aldeppenmeier , @dmacias , @ngonzalez , @dvolpi , @vguemas , @pabretonniere
This issue has been opened as thread to discuss obstacles and solutions in relation with usage of masks in s2dverification after some claimed it is usually a headache.
A short time ago a security check was added to Load() to ensure that any provided mask had the expected size (see issue #46 ) and it was suggested that there should be an option to apply masks before interpolating into the common grid if any (see issue #59 ).
But still there are other relevant issues:
- There should be a mechanism to automatically associate a dataset to its corresponding mask stored in file system.
- There should be a convention or mechanism to make sure masks are always loaded with the longitudes and latitudes in the right order and so make the usage of masks more reliable.
What's been your experience so far with masks in s2dverification? Would you comment any other issue?
Would you suggest any solution to these problems?
Any feedback is of course very much appreciated.