s2dverification issueshttps://earth.bsc.es/gitlab/es/s2dverification/-/issues2015-02-25T12:02:23+01:00https://earth.bsc.es/gitlab/es/s2dverification/-/issues/18Fortran version Mean1Dim2015-02-25T12:02:23+01:00Carles Carmona-CalpeFortran version Mean1DimFortran version Mean1DimFortran version Mean1DimCarles Carmona-CalpeCarles Carmona-Calpehttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/20Some comments lost in merging v2.3.0 into develop-ACC_permutation2015-07-27T01:06:47+02:00Nicolau Manubens GilSome comments lost in merging v2.3.0 into develop-ACC_permutationI have seen that at least some additional code comments in Load.R have been lost when performing this merge
20a0071b089352bccc6d4fa5639880f7c8b42756
I should recheck carefully (huge diff) to see what has been lost and what branch it ca...I have seen that at least some additional code comments in Load.R have been lost when performing this merge
20a0071b089352bccc6d4fa5639880f7c8b42756
I should recheck carefully (huge diff) to see what has been lost and what branch it came from.Nicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/54MeanListDim(): documentation enhancement and extra parameters2015-08-04T17:33:40+02:00Nicolau Manubens GilMeanListDim(): documentation enhancement and extra parametersThe documentation of this function should explain that the average is applied over the specified dimensions in reverse order, not over all dimensions at the same time.
Moreover, extra parameters should be added to be able to choose a ...The documentation of this function should explain that the average is applied over the specified dimensions in reverse order, not over all dimensions at the same time.
Moreover, extra parameters should be added to be able to choose a dimension order or to perform the average over all the dimensions at the same time.Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/56Load(): Make it able to load members stored in various formats2015-04-22T19:16:11+02:00Nicolau Manubens GilLoad(): Make it able to load members stored in various formatsCurrently the only supported member file name is a path with the member number ranging from 0 to nmemb - 1. But in some cases it ranges from 1 to nmemb or contains more padding 0s (0001, 0002, ...) or is in the format rNiMpO, where N, M ...Currently the only supported member file name is a path with the member number ranging from 0 to nmemb - 1. But in some cases it ranges from 1 to nmemb or contains more padding 0s (0001, 0002, ...) or is in the format rNiMpO, where N, M and O can vary.
A solution has to be thought and coded.Nicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/59Load(): Option to apply mask before interpolating2015-04-24T15:28:25+02:00Nicolau Manubens GilLoad(): Option to apply mask before interpolatingAs discussed in issue #48:
@eexarchou said:
Thank you Nico for finding this problem.
Indeed if there lonlat data, there should be one common grid. However, sometimes it is important that the mask is applied before regridding ("c...As discussed in issue #48:
@eexarchou said:
Thank you Nico for finding this problem.
Indeed if there lonlat data, there should be one common grid. However, sometimes it is important that the mask is applied before regridding ("cdo remapcon …") because in ocean data, there are zeros or other values over land points, that should not be taken into account during interpolation. This is not the case for atmospheric data. I am not sure now how this can be implemented, but I think it is important for points near the cost.
Cheers Eleftheria
@cprodhomme said:
Hi,
You are right Eleftheria that the mask should be use before the interpolation (normally for tos, the file itself is masked so it won't generate any issue). However, it we want to apply the mask before the interpolation, it will be needed to specify the mask of the observation always in the Load function, which will break the backward compatibility of Load.... It we decide that we want to go for this major change, it would be a good opportunity to improve in general the mask handling of load (check if the size is compatible, use directly the name of the file, or even automatic mask selection?) This should be discuss I think in an s2dverification meeting, no?
Regards.
Chloé
@vguemas said:
I agree that it would be a very interesting enhancement to apply the mask before interpolation rather than after interpolation as it is done currently. The current Load assumes that the initial data are already land/sea masked and the application of an additional mask is rather thought to discard additional values that are not of interest for a specific analysis. However, there might be several application where this assumption does not hold: interpolation of winds over ocean without being poluted by land winds ... etc
I don't see clearly though why there would be automatically a backward compatibility break if modifying the Load behaviour : there could be an additional option to apply the mask before interpolation. I think in any case you need to apply the mask again after interpolation because the input mask defined on various grids would not necessarily give the same output masks even though they intend to define the same geographical boundaries initially. I mean for a same given resolution, there exists different land/sea masks depending on the dataset considered even though there is only one geographical location of the continents.
That'd be nice to have a check of the size compatibility, the direct use of the path/name of the mask. However, I am not sure an automatic selection of the mask is suitable since there might be other applications of masking than the classical land/sea mask.
Nicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/60Load(): Modifications to load CMOR files2015-07-23T15:15:19+02:00Nicolau Manubens GilLoad(): Modifications to load CMOR filesThese modifications are needed urgently:
- Parameter / configuration file field to choose the names of the dimensions in the .nc files
- Modify nmember parameter or add a new parameter / config file field to choose the pattern of the m...These modifications are needed urgently:
- Parameter / configuration file field to choose the names of the dimensions in the .nc files
- Modify nmember parameter or add a new parameter / config file field to choose the pattern of the member name: rNiMpO
- Make Load() able to read chunks in separate filesNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/62issue with heatmap2015-05-18T19:27:38+02:00Louis-Philippe Caronissue with heatmapfor function heatmap.R in develop-heatmap:
I'm having 2 problems:
1) when I create a figure, I have a white page before. Any idea how I can remove it?
2) when the plot is too big, the legend gets hidden. Do you know what should ...for function heatmap.R in develop-heatmap:
I'm having 2 problems:
1) when I create a figure, I have a white page before. Any idea how I can remove it?
2) when the plot is too big, the legend gets hidden. Do you know what should be done to avoid this type of situation?
Here is the code I use to test the function:
figpath='~/Desktop/TEMP/'
heatmapfile=paste(figpath,'heatmap.eps',sep='')
ndata=9
nrow1=3
ncol1=ndata/nrow1
mat.pval <- matrix(runif(ndata, min=0.01, max=0.25), nrow=nrow1)
mat.beta <- matrix(rnorm(ndata), nrow=nrow1)
mat.pval[1,1] <- 0.001
#rownames(mat.beta) <- rownames(mat.pval) <- c("A", "B", "C")
#colnames(mat.beta) <- colnames(mat.pval) <- c("X", "Y", "Z")
postscript(heatmapfile)
heatmap(mat.pval=mat.pval, mat.beta=mat.beta, pos_leg=c(0.6,0))
dev.off()Nicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/63Load(): Using masks is sometimes too complex.2015-06-02T01:37:49+02:00Nicolau Manubens GilLoad(): Using masks is sometimes too complex.@cprodhomme , @vtorralba , @mmenegoz , @lpcaron , @lbatte , @obellprat , @nevensf , @eexarchou , @fmassonnet , @aldeppenmeier , @dmacias , @ngonzalez , @dvolpi , @vguemas , @pabretonniere
This issue has been opened as thread to discu...@cprodhomme , @vtorralba , @mmenegoz , @lpcaron , @lbatte , @obellprat , @nevensf , @eexarchou , @fmassonnet , @aldeppenmeier , @dmacias , @ngonzalez , @dvolpi , @vguemas , @pabretonniere
This issue has been opened as thread to discuss obstacles and solutions in relation with usage of masks in s2dverification after some claimed it is usually a headache.
A short time ago a security check was added to Load() to ensure that any provided mask had the expected size (see issue #46 ) and it was suggested that there should be an option to apply masks before interpolating into the common grid if any (see issue #59 ).
But still there are other relevant issues:
- There should be a mechanism to automatically associate a dataset to its corresponding mask stored in file system.
- There should be a convention or mechanism to make sure masks are always loaded with the longitudes and latitudes in the right order and so make the usage of masks more reliable.
What's been your experience so far with masks in s2dverification? Would you comment any other issue?
Would you suggest any solution to these problems?
Any feedback is of course very much appreciated.https://earth.bsc.es/gitlab/es/s2dverification/-/issues/64Load(): Bug in s2dv >= 2.3.0 when order of variable definitions in NetCDF fil...2016-02-24T18:37:01+01:00Nicolau Manubens GilLoad(): Bug in s2dv >= 2.3.0 when order of variable definitions in NetCDF files is changed. Problems with infinite values.After e-mail exchange and meetings between @mmenegoz , @pabretonniere and I, Martin has detected that when the variable definitions in the NetCDF files are changed of order, Load() may crash.
This didn't happen in s2dv < 2.3.0 which r...After e-mail exchange and meetings between @mmenegoz , @pabretonniere and I, Martin has detected that when the variable definitions in the NetCDF files are changed of order, Load() may crash.
This didn't happen in s2dv < 2.3.0 which reminds of a bug or change of behaviour in ncdf4 R library.
Martin's examples:
This is a temporary experiment output with only some members:
```
/cfunas/exp/MORDICUS/PTUBO_1991/save_monthly_mean/tas_6hourly/tas_19910101.nc
```
This is the final file with all the members, after PA concatenated them to the original file:
```
/cfunas/exp/MORDICUS/PTUBO_1991/monthly_mean/tas_6hourly/tas_19910101.nc
```
This is the path to the script to load the data:
```
/cfunas/scratch/mmenegoz/MORDICUS/map_tas_mordicus_summer.R
```
This is the output of the script:
```
[1] "* /cfunas/exp/MORDICUS/PTUBO_1991/monthly_mean/tas_6hourly/tas_19910101.nc"
starting worker pid=21305 on localhost:10187 at 17:27:18.135
starting worker pid=21314 on localhost:10187 at 17:27:18.380
starting worker pid=21323 on localhost:10187 at 17:27:18.638
starting worker pid=21332 on localhost:10187 at 17:27:18.876
starting worker pid=21342 on localhost:10187 at 17:27:19.113
starting worker pid=21351 on localhost:10187 at 17:27:19.358
starting worker pid=21360 on localhost:10187 at 17:27:19.605
starting worker pid=21369 on localhost:10187 at 17:27:19.840
cdf_put_vara_double : ncid = 131072 varid = 5 val0 = inf
cdf_put_vara_double : varname = tas
Error in checkForRemoteErrors(val) :
one node produced an error: incorrect number of dimensions
```Nicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/66problem when scratch is full with Load2015-06-18T18:49:23+02:00Chloé Prodhommeproblem when scratch is full with LoadHi,
I may look a bit stupid but I just lost a huge amount of time trying to understand what was wrong with the new load, compare to the old one, have a look at the file, changes the arguments... and the error I got where changing and ...Hi,
I may look a bit stupid but I just lost a huge amount of time trying to understand what was wrong with the new load, compare to the old one, have a look at the file, changes the arguments... and the error I got where changing and at some point I was starting to get crazy and I realized that the issue was that the scratch was full si it was failing.
If it is not too hard, maybe @nmanubens you could had some error message for this case (which probably going to happen quite a lot as long as people will be working on daily data).
BTW, as I am running old script and I was trying to find the bug I compared the new and the old load version, and the new is SO SO much faster! Thanks a lot for that!
Best regards.
Chloé
Nicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/67Load(): warning messages about ncdf and about socket connections. ncdf4 bug.2015-07-03T15:33:13+02:00Nicolau Manubens GilLoad(): warning messages about ncdf and about socket connections. ncdf4 bug.Load() uses cdo tools to regrid and ncrename to rename files at various points along its execution.
In some cases, for example when issuing the following script
```
Data <- Load(exp = c("m04o", "m04p"), obs = "ESA", var = "tos", sda...Load() uses cdo tools to regrid and ncrename to rename files at various points along its execution.
In some cases, for example when issuing the following script
```
Data <- Load(exp = c("m04o", "m04p"), obs = "ESA", var = "tos", sdates = paste0(1993:2009, '0501'), nleadtime = 4, leadtimemin = 1, leadtimemax = 4, storefreq = "monthly", sampleperiod = 1, nmember = 10, output = "lonlat", lonmin = -170, lonmax = -120, latmin = -5, latmax = 5, configfile="~obellprat/CFU/s2dverification/inst/config/IC3.conf")
```
the following warnings raise repeatedly (Execution doesn't stop, actually results are obtained properly):
```
Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
Execution halted
```
```
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)
```
I think the first kind of errors are caused because of the second kind of errors.
Testing the cdo and ncrename commands manually
```
cdo remapcons,t106grid -selname,tos /cfu/data/esa/monthly_mean/tos/tos_200908.nc tmp.nc
ncrename -v lat,latitude -d lat,latitude -v lon,longitude -d lon,longitude -v .lev,ensemble -d .lev,ensemble -d .reftime,time tmp.nc
```
I get the following error output:
```
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_enddef()
nco_err_exit(): ERROR Error code is -101. Translation into English with nc_strerror(-101) is "NetCDF: HDF error"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)
```
After checking in the Internet I've found it's likely an ncdf error, which is fixed in versions beyond 4.3.1 (see http://sourceforge.net/p/nco/bugs/44/)Nicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/69Load(): Cdo remap time could be improved2015-06-27T13:59:25+02:00Nicolau Manubens GilLoad(): Cdo remap time could be improvedAs seen in page 109 of https://www.rsmas.miami.edu/users/rajib/cdo.pdf, remap weights could be computed only once and then re-used so as to save huge amount of time.As seen in page 109 of https://www.rsmas.miami.edu/users/rajib/cdo.pdf, remap weights could be computed only once and then re-used so as to save huge amount of time.Nicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/71Load(): Loading observations only bug2015-07-06T19:13:54+02:00Nicolau Manubens GilLoad(): Loading observations only bugVerónica reported that when you specify a starting date far back in time, the loaded lead-times go beyond the current date. As she detected, that's due to a bad calculation of months.
She'll correct it if she has free time anytime soon.Verónica reported that when you specify a starting date far back in time, the loaded lead-times go beyond the current date. As she detected, that's due to a bad calculation of months.
She'll correct it if she has free time anytime soon.Verónica Torralba-FernándezVerónica Torralba-Fernándezhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/72System call error output is ignored2015-07-09T10:50:56+02:00Nicolau Manubens GilSystem call error output is ignoredIt is unsafe to inore the system calls error output. In some cases these will crash and will generate unpercieved wrong data.It is unsafe to inore the system calls error output. In some cases these will crash and will generate unpercieved wrong data.Nicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/73RMSE bootstrap members2015-07-13T12:18:11+02:00Daniel Macias GómezRMSE bootstrap membershttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/74LoadError function for generating new observations by bootstrapping2015-07-13T13:50:53+02:00Daniel Macias GómezLoadError function for generating new observations by bootstrappinghttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/75Load(): It is not able to load data sliced in earth sub-regions2015-07-23T15:14:10+02:00Nicolau Manubens GilLoad(): It is not able to load data sliced in earth sub-regionsNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/78Filter()'s name should be changed so that it doesn't collide with base's Filt...2015-07-27T13:53:07+02:00Nicolau Manubens GilFilter()'s name should be changed so that it doesn't collide with base's Filter() That's required as per CRAN policiesThat's required as per CRAN policiesRelease 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/81Load(): Allow arithmetics in configfile expressions2015-07-31T10:12:16+02:00Nicolau Manubens GilLoad(): Allow arithmetics in configfile expressionsNicolau Manubens GilNicolau Manubens Gilhttps://earth.bsc.es/gitlab/es/s2dverification/-/issues/86ncdf vs. ncdf4 or both?2015-08-19T16:58:11+02:00Nicolau Manubens Gilncdf vs. ncdf4 or both?As seen in https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CC4QFjACahUKEwj0mrmW_LTHAhULvBQKHarlBK4&url=https%3A%2F%2Fpublicwiki.deltares.nl%2Fdownload%2Fattachments%2F42401998%2FFedorBaart_ncdf4.pdf%3Fversion%3D1%26m...As seen in https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CC4QFjACahUKEwj0mrmW_LTHAhULvBQKHarlBK4&url=https%3A%2F%2Fpublicwiki.deltares.nl%2Fdownload%2Fattachments%2F42401998%2FFedorBaart_ncdf4.pdf%3Fversion%3D1%26modificationDate%3D1286292950000&ei=wF7UVfS6JIv4UqrLk_AK&usg=AFQjCNFChZotf4Z5RgzP_xINhyg1kg6vYQ&sig2=SikGlu6X7pTCIkVh5FuzZA&bvm=bv.99804247,d.d24
ncdf4 breaks windows capabilities whereas ncdf doesn't. Moreover, ncdf is able to load OPeNDAP files too. Should we consider moving back to ncdf or adding ncdf alongside ncdf4 so as not to lose windows compatibility?Release 3.0.0 to CRANNicolau Manubens GilNicolau Manubens Gil