s2dverification issueshttps://earth.bsc.es/gitlab/es/s2dverification/-/issues2020-10-07T19:10:05+02:00https://earth.bsc.es/gitlab/es/s2dverification/-/issues/169Load(): Crashing randomly when working on SMP2020-10-07T19:10:05+02:00Nicolau Manubens GilLoad(): Crashing randomly when working on SMPLoad() was used several times. 20% of the times it crashed, but running it again with the same parameters would eventually work. As reported by @llledo .
```r
* The load call you issued is:
* Load(var = "va60ml", exp = list(struct...Load() was used several times. 20% of the times it crashed, but running it again with the same parameters would eventually work. As reported by @llledo .
```r
* The load call you issued is:
* Load(var = "va60ml", exp = list(structure(list(name = "eraint", path =
* "/esnas/reconstructions/ecmwf/$EXP_NAME$/6hourly/$VAR_NAME$/$VAR_NAME$_$YEAR$$MONTH$.nc"),
* .Names = c("name", "path"))), obs = NULL, sdates = "19941001",
* grid = NULL, output = "lonlat", storefreq = "monthly", ...)
* See the full call in '$load_parameters' after Load() finishes.
* Fetching first experimental files to work out 'var_exp' size...
* Exploring dimensions...
* /esnas/reconstructions/ecmwf/eraint/6hourly/va60ml/va60ml_199410.nc
* Success. Detected dimensions of experimental data: 1, 1, 1, 124, 256,
* 512
* Fetching first observational files to work out 'var_obs' size...
* Success. Detected dimensions of observational data: 0
* Will now proceed to read and process 1 data files:
* /esnas/reconstructions/ecmwf/eraint/6hourly/va60ml/va60ml_199410.nc
* Total size of requested data: 130023424 bytes.
* - Experimental data: ( 1 x 1 x 1 x 124 x 256 x 512 ) x 8 bytes =
* 130023424 bytes.
* - Observational data: ( 0 ) x 8 bytes = 0 bytes.
* If size of requested data is close to or above the free shared RAM
* memory, R will crash.
Error in attach.resource(obj, ...) :
Fatal error in attach: big.matrix could not be attached.
Error: dims [product 16252928] do not match the length of object [0]
Execution halted
```Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/163Probelm of PlotEquiMap() when plotting data in certain orders2017-10-09T12:52:29+02:00Omar BellpratProbelm of PlotEquiMap() when plotting data in certain ordersHi @nmanubens ,
I get a strange grid when loading
/esnas/obs/ukmo/hadcrut_v4/monthly_mean/tas/
as you can see in the attached figure. The load script is
/home/Earth/obellpra/R/bams_2015/monthly_hadcrut4.r
[february_anom_h...Hi @nmanubens ,
I get a strange grid when loading
/esnas/obs/ukmo/hadcrut_v4/monthly_mean/tas/
as you can see in the attached figure. The load script is
/home/Earth/obellpra/R/bams_2015/monthly_hadcrut4.r
[february_anom_hadcrut4_tas.pdf](https://earth.bsc.es/gitlab/es/s2dverification/uploads/f04ec0f75727dadfd11c1d87ab0041e7/february_anom_hadcrut4_tas.pdf)
Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/140High performance and scores2019-11-11T18:23:13+01:00Nicolau Manubens GilHigh performance and scoresCurrently, computation of scores or other diagnostics in s2dverification is slow.
It is needed to optimize the functions in terms of:
a) Computing time, for example by using base R vectorised functions or by calling C functions
...Currently, computation of scores or other diagnostics in s2dverification is slow.
It is needed to optimize the functions in terms of:
a) Computing time, for example by using base R vectorised functions or by calling C functions
b) Memory usage, for example by avoiding unneeded copies of arrays in the score functions.
c) Parallelism, particularly making them able to run on clusters with thousands of cores.
@ncortesi already programmed a set of scripts that allow to run score functions by chunks on MareNostrum (or SMP, with less than 128 tasks) and analysed the performance, giving a solution to c). More information can be found at https://earth.bsc.es/wiki/lib/exe/fetch.php?media=library:internal:20160615_running_diagnostics_on_marenostrum.pdf . This solution is, however, not ready to be used without some difficult configuration steps. @nmanubens is in contact with him to solve this issue. Additionally, in the long term, this solution should ideally be ported to R and make it available through parameters in the typical score functions or by evolving the `veriApply()` function in the package easyVerification.
@nmanubens and @jginer are working on a compatibility break in s2dverification in which the data model used throughout will be revised and improved taking into account existing libraries that, for example, offer abstractions of distributed arrays in a way that, a simple sum of two arrays, is launched transparently and automatically in the multiple nodes. Furthermore, this model has to be agreed and common with the one in downscaleR (discussion ongoing with JoaquĆn from UNICAN) and it should also end up being, ideally, the data model used in QA4Seas.
The Data & Diagnostics team is also investigating existing paradigms and techniques to deal with this kind of problems.Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/139Load(): Crashes without warning when you request for a time step that is not ...2016-06-06T17:08:45+02:00Nicolau Manubens GilLoad(): Crashes without warning when you request for a time step that is not in a file-per-dataset.Example:
```r
data <- Load('heatc', NULL, 'glorys2_v1', '19901101', leadtimemax = 1)
```Example:
```r
data <- Load('heatc', NULL, 'glorys2_v1', '19901101', leadtimemax = 1)
```Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/133AA() from eigentechniques can now be integrated into s2dverification2016-06-02T17:05:43+02:00Nicolau Manubens GilAA() from eigentechniques can now be integrated into s2dverificationRelease 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/107Load(): Unable to load a single lonlat point.2020-04-17T18:17:12+02:00Nicolau Manubens GilLoad(): Unable to load a single lonlat point.```
datap <- Load('tos', 'i00k', NULL, '19901101', output = 'lonlat', leadtimemax = 1, lonmin = 0.5, lonmax = 0.5, latmin = 89.5, latmax = 89.5)
```
crashes
```
datap <- Load('tos', 'i00k', NULL, '19901101', output = 'lonlat', lea...```
datap <- Load('tos', 'i00k', NULL, '19901101', output = 'lonlat', leadtimemax = 1, lonmin = 0.5, lonmax = 0.5, latmin = 89.5, latmax = 89.5)
```
crashes
```
datap <- Load('tos', 'i00k', NULL, '19901101', output = 'lonlat', leadtimemax = 1, lonmin = 0, lonmax = 1.12, latmin = 88, latmax = 89.1)
```
crashes in develop-THREDDS-subsetting
```
datap <- Load('tos', 'i00k', NULL, '19901101', output = 'lonlat', leadtimemax = 1, lonmin = 0, lonmax = 1.13, latmin = 88, latmax = 89.2)
```
works but gives a 2x2 matrixRelease 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/106Enlarge(): Is highly unefficient. Should be rewritten/removed.2015-10-08T21:10:03+02:00Nicolau Manubens GilEnlarge(): Is highly unefficient. Should be rewritten/removed.Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/99Score and statistics documentation enhancement.2015-09-28T12:19:05+02:00Nicolau Manubens GilScore and statistics documentation enhancement.Most statistics and scores receive 'exp' and 'obs' as parameters and this can make the user think only experimental data from Load() can be provided to 'exp' and only observational data can be provided to 'obs', while both data arrays fo...Most statistics and scores receive 'exp' and 'obs' as parameters and this can make the user think only experimental data from Load() can be provided to 'exp' and only observational data can be provided to 'obs', while both data arrays follow the same structure and could be provided interchangeably.Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/97Load(): Subsetting along members is not possible.2015-11-15T21:01:49+01:00Nicolau Manubens GilLoad(): Subsetting along members is not possible.Currently, Load() has the parameters 'nmember' and 'nmemberobs' which allow to specify a number of members for each experiment and observation, but doesn't allow to pick which members to load, so all must be loaded.
We could code the ...Currently, Load() has the parameters 'nmember' and 'nmemberobs' which allow to specify a number of members for each experiment and observation, but doesn't allow to pick which members to load, so all must be loaded.
We could code the following behaviour for 'nmember':
- If it receives a single integer N, it replicates it as many times as experiments in 'exp', and will mean that N members will be loaded from each.
- If it receives a vector with different integers, different numbers of members will be loaded from each experiment.
- If it receives a list of vectors, the specified members are loaded.
Same for 'nmemberobs' and 'obs'.Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/86ncdf vs. ncdf4 or both?2015-08-19T16:58:11+02:00Nicolau Manubens Gilncdf vs. ncdf4 or both?As seen in https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CC4QFjACahUKEwj0mrmW_LTHAhULvBQKHarlBK4&url=https%3A%2F%2Fpublicwiki.deltares.nl%2Fdownload%2Fattachments%2F42401998%2FFedorBaart_ncdf4.pdf%3Fversion%3D1%26m...As seen in https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CC4QFjACahUKEwj0mrmW_LTHAhULvBQKHarlBK4&url=https%3A%2F%2Fpublicwiki.deltares.nl%2Fdownload%2Fattachments%2F42401998%2FFedorBaart_ncdf4.pdf%3Fversion%3D1%26modificationDate%3D1286292950000&ei=wF7UVfS6JIv4UqrLk_AK&usg=AFQjCNFChZotf4Z5RgzP_xINhyg1kg6vYQ&sig2=SikGlu6X7pTCIkVh5FuzZA&bvm=bv.99804247,d.d24
ncdf4 breaks windows capabilities whereas ncdf doesn't. Moreover, ncdf is able to load OPeNDAP files too. Should we consider moving back to ncdf or adding ncdf alongside ncdf4 so as not to lose windows compatibility?Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/78Filter()'s name should be changed so that it doesn't collide with base's Filt...2015-07-27T13:53:07+02:00Nicolau Manubens GilFilter()'s name should be changed so that it doesn't collide with base's Filter() That's required as per CRAN policiesThat's required as per CRAN policiesRelease 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/54MeanListDim(): documentation enhancement and extra parameters2015-08-04T17:33:40+02:00Nicolau Manubens GilMeanListDim(): documentation enhancement and extra parametersThe documentation of this function should explain that the average is applied over the specified dimensions in reverse order, not over all dimensions at the same time.
Moreover, extra parameters should be added to be able to choose a ...The documentation of this function should explain that the average is applied over the specified dimensions in reverse order, not over all dimensions at the same time.
Moreover, extra parameters should be added to be able to choose a dimension order or to perform the average over all the dimensions at the same time.Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gil