Earth Diagnostics issueshttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues2024-01-22T10:38:15+01:00https://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/145Order of dimensions changed for heatsum2024-01-22T10:38:15+01:00Pierre-Antoine BretonnièreOrder of dimensions changed for heatsum2 files generated with the same config file but for different dates have a different order of dimension, preventing them from properly concatenating in the monitor job.
Discovered in t0e7:
```
pbretonn@bscearth319:/esarchive/exp/eceart...2 files generated with the same config file but for different dates have a different order of dimension, preventing them from properly concatenating in the monitor job.
Discovered in t0e7:
```
pbretonn@bscearth319:/esarchive/exp/ecearth/t0e7/diags/DCPP/EC-Earth-Consortium/EC-Earth3/dcppA-hindcast/r9i1p1f1$ ncdump -h ./Omon/heatcsum0-700m/gn/v20240116/heatcsum0-700m_Omon_EC-Earth3_dcppA-hindcast_s2001-r9i1p1f1_gn_200101-200112.nc
netcdf heatcsum0-700m_Omon_EC-Earth3_dcppA-hindcast_s2001-r9i1p1f1_gn_200101-200112 {
dimensions:
region = 5 ;
time = 12 ;
region_length = 15 ;
variables:
double time(time) ;
time:axis = "T" ;
time:units = "days since 1850-01-01 00:00:00" ;
time:standard_name = "time" ;
time:long_name = "time" ;
time:calendar = "proleptic_gregorian" ;
double heatcsum0-700m(**region, time**) ;
heatcsum0-700m:long_name = "Total Ocean heat content" ;
heatcsum0-700m:units = "J" ;
heatcsum0-700m:coordinates = "region time" ;
heatcsum0-700m:standard_name = "total_ocean_heat_content" ;
heatcsum0-700m:_FillValue = 1.e+20 ;
heatcsum0-700m:missingValue = 1.e+20 ;
char region(region, region_length) ;
region:standard_name = "region" ;
// global attributes:
:invalid_standard_name = "total_ocean_heat_content" ;
:missingValue = 1.e+20 ;
:modeling_realm = "ocean" ;
:table_id = "Table Omon (December 2013)" ;
:Conventions = "CF-1.7" ;
:NCO = "netCDF Operators version 4.9.9 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
:history = "UTC 2024-01-18T07:30:49.514896: Diagnostic Heat content layer Startdate: 20010101 Member: 9 Chunk: 1 Box: 0-700m calculated with EarthDiagnostics version 3.5.9;UTC 2024-01-18T07:30:49.512999: Using Basins masks file version with grid Ec3.2_O1L75. Original file can be found in /esarchive/autosubmit/conf_files.;Thu Jan 18 08:30:49 2024: /gpfs/projects/bsc32/software/suselinux/11/software/earthdiagnostics/3.5.9/bin/ncatted -O -a _FillValue,heatcsum0-700m,o,d,1.e20 -a missingValue,heatcsum0-700m,o,d,1.e20 --nco_dbg_lvl=3 --output=/scratch/tmp/2730916/diags/t0e7/templuj2p2cc.nc /scratch/tmp/2730916/diags/t0e7/templuj2p2cc.nc\nThu Jan 18 08:30:49 2024: /gpfs/projects/bsc32/software/suselinux/11/software/earthdiagnostics/3.5.9/bin/ncatted -O -a _FillValue,heatcsum0-700m,o,d,1.e20 -a missingValue,heatcsum0-700m,o,d,1.e20 --nco_dbg_lvl=3 --output=/scratch/tmp/2730916/diags/t0e7/templuj2p2cc.nc /scratch/tmp/2730916/diags/t0e7/templuj2p2cc.nc" ;
:_NCProperties = "version=2,netcdf=4.7.4,hdf5=1.10.6" ;
}
(base) pbretonn@bscearth319:/esarchive/exp/ecearth/t0e7/diags/DCPP/EC-Earth-Consortium/EC-Earth3/dcppA-hindcast/r9i1p1f1$ ncdump -h ./Omon/heatcsum0-700m/gn/v20240116/heatcsum0-700m_Omon_EC-Earth3_dcppA-hindcast_s2001-r9i1p1f1_gn_200201-200212.nc
netcdf heatcsum0-700m_Omon_EC-Earth3_dcppA-hindcast_s2001-r9i1p1f1_gn_200201-200212 {
dimensions:
time = 12 ;
region = 5 ;
region_length = 50 ;
variables:
double time(time) ;
time:bounds = "time_bnds" ;
time:units = "days since 1850-01-01 00:00:00" ;
time:calendar = "proleptic_gregorian" ;
time:axis = "T" ;
time:long_name = "time" ;
time:standard_name = "time" ;
char region(region, region_length) ;
double heatcsum0-700m(**time, region**) ;
heatcsum0-700m:units = "J" ;
heatcsum0-700m:coordinates = "region time" ;
heatcsum0-700m:standard_name = "total_ocean_heat_content" ;
heatcsum0-700m:long_name = "Total Ocean heat content" ;
heatcsum0-700m:_FillValue = 1.e+20 ;
heatcsum0-700m:missingValue = 1.e+20 ;
// global attributes:
:modeling_realm = "ocean" ;
:table_id = "Table Omon (December 2013)" ;
:NCO = "netCDF Operators version 4.9.9 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
:history = "UTC 2024-01-18T07:16:40.625440: Diagnostic Heat content layer Startdate: 20010101 Member: 9 Chunk: 2 Box: 0-700m calculated with EarthDiagnostics version 3.5.9;UTC 2024-01-18T07:16:40.623510: Using Basins masks file version with grid Ec3.2_O1L75. Original file can be found in /esarchive/autosubmit/conf_files.;Thu Jan 18 08:16:40 2024: /gpfs/projects/bsc32/software/suselinux/11/software/earthdiagnostics/3.5.9/bin/ncatted -O -a _FillValue,heatcsum0-700m,o,d,1.e20 -a missingValue,heatcsum0-700m,o,d,1.e20 --nco_dbg_lvl=3 --output=/scratch/tmp/2730895/diags/t0e7/tempw6g88lus.nc /scratch/tmp/2730895/diags/t0e7/tempw6g88lus.nc" ;
:_NCProperties = "version=2,netcdf=4.7.4,hdf5=1.10.6" ;
```
Rerunning the job fixes it (to be confirmed by @eferre1 ) and the job is anyway being replaced by ESMValTool by @sghosh but commenting it here for proper documentationhttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/144Python version update in docs?2022-09-07T18:15:58+02:00Eneko Martineneko.martin@bsc.esPython version update in docs?As I can see in [environment.yml](https://earth.bsc.es/gitlab/es/earthdiagnostics/-/blob/master/environment.yml) Python 3.x is supported (it must be lower than 3.9). However, the documentation says that it only works with Python 2.x ([se...As I can see in [environment.yml](https://earth.bsc.es/gitlab/es/earthdiagnostics/-/blob/master/environment.yml) Python 3.x is supported (it must be lower than 3.9). However, the documentation says that it only works with Python 2.x ([see installation requirements](https://earthdiagnostics.readthedocs.io/en/stable/tutorial.html#installation)). Then, I suppose that the required Python version should be updated in docs.https://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/143Homogenization of basins files used by Earthdiagnostics2023-04-27T15:05:35+02:00acarreriHomogenization of basins files used by EarthdiagnosticsHi Earthdiagnostics potential users, @vlapin, @rfernand, @vsicardi , @jacosta , @yruprich , @eexarchou , @emoreno, @etourign
hi @mcastril , @eferre1,
Continuing in the line of trying to homogenize our diagnoses, I raise this issue abo...Hi Earthdiagnostics potential users, @vlapin, @rfernand, @vsicardi , @jacosta , @yruprich , @eexarchou , @emoreno, @etourign
hi @mcastril , @eferre1,
Continuing in the line of trying to homogenize our diagnoses, I raise this issue about the basin masks used in Earthdiagnostics, which is now implemented in the workflow of auto-ecearth.
As you may know, you can ask for diagnostics (`regmean` but also `moc`) to be computed in specific regions.
Earthdiagnostics uses for that the `basins.Ec${version}_${RES}.nc` in `/esarchive/autosubmit/con_files/`.
Right now, there is:
- basins.Ec3.2_O1L75.nc
- basins.Ec3.2_O25L75.nc
- basins.Ec3.6_12L75.nc (that @emoreno recently produced).
**BUT** the different regions are not consistent between grids!
For instance, the first region, named `AMV_North_Atlantic` looks like, depending on the grid (from ORCA1 to ORCA012): ![Screenshot_from_2022-02-23_13-39-06](/uploads/052be7f342ecb3eed06d9347c1fe0ca3/Screenshot_from_2022-02-23_13-39-06.png)
These `basins` files are produced by this [script](https://earth.bsc.es/gitlab/cp/cpg-shared-tools/-/blob/master/New%20Diagnostics/Climate%20Indices/Basin%20and%20Regional%20Masks/create_basins_mask.py) and uses the [basins.yml](https://earth.bsc.es/gitlab/cp/cpg-shared-tools/-/tree/master/New%20Diagnostics/Climate%20Indices/Basin%20and%20Regional%20Masks) for the definition of the regions.
Three things here:
- the regions should be consistent between grids except that this `basins.yml` file uses for the main oceans (actually only `Atlantic_Ocean` is called in the `basins.yml` file) the basins from Nemo directly, that is from this file: `/esarchive/autosubmit/con_files/new_maskglo.Ec3.[Screenshot_from_2022-02-23_14-56-20](/uploads/be909a85f752ac2f602bb4ffc06307c9/Screenshot_from_2022-02-23_14-56-20.png)
- some regions such as `AMV_North_Atlantic` may have been created in a different way for ORCA1 and ORCA025, that's why it's different from the one in ORCA12 (which uses the correct definition in `basins.yml`).
- some regions already included in ORCA1 and ORCA025 are not present in the new ORCA12 file (`basins.Ec3.6_12L75.nc`), because these regions haven't been updated in our `basins.yml` in cpg_shared_tool project. This file shows the different regions present or not: [Earthdiagnostics_regions.ods](/uploads/f311c87904d4be3dcf37b45f667e6520/Earthdiagnostics_regions.ods)
In any case, it can be the moment to redefine/clean the regions we want. Also because for instance, when calling for the 'moc' function in Earthdiagnostics, if you call it for the `North_Atlantic_Ocean` region, you compute it over the North Atlantic + the whole Arctic in ORCA1 and ORCA12, over only the North Atlantic (not the highest latitudes) in ORCA025 when we may want the North_Atlantic + the extension of the Arctic only in the Atlantic part, not including the other Pacific part of the Arctic.
It can be done easily:
- we have to clean or update the basins.yml file
- we have to check the `new_maskglo.{version}_{res}.nc` file for the latest *3.6* version of NEMO and see if they are coherent between resolution (I think the issue is only for ORCA025 here).
- we have to produce the basins.Ec{version}_{res}.nc for this latest 3.6 version with the `create_basins_mask.py`
Who's with me?
PYI @portegaacarreriacarrerihttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/142Issue with a mask?2022-02-25T10:03:44+01:00acarreriIssue with a mask?Hi @sloosvel ,@pabretonniere ,
@pabretonniere, I finally successfully launched the earthdiagnostics jobs with auto-ecearth (reducing a lot the number of diagnostics).
The first two jobs, (for the first two chunks) succeeded but for the...Hi @sloosvel ,@pabretonniere ,
@pabretonniere, I finally successfully launched the earthdiagnostics jobs with auto-ecearth (reducing a lot the number of diagnostics).
The first two jobs, (for the first two chunks) succeeded but for the third one, I have this error in the .out file (`/esarchive/autosubmit/a4a0/tmp/LOG_a4a0/a4a0_19800101_fc00_3_EARTHDIAGS.20220215152227.out`):
~~~
^[[31m[ERROR] Interpolate with CDO Startdate: 19800101 Member: 0 Chunk: 3 Variable: ocean:tos Frequency: mon Target grid: r360x180 Original grid: Mask ocean: True Model: Ec3.2_O25L75: [Errno -101] NetCDF: HDF error: b'mask.nc'
File "/opt/conda/lib/python3.8/site-packages/earthdiagnostics/work_manager.py", line 313, in _run_job
job.compute()
File "/opt/conda/lib/python3.8/site-packages/earthdiagnostics/ocean/interpolatecdo.py", line 318, in compute
mask = Utils.get_mask(Basins().Global).astype(float)
File "/opt/conda/lib/python3.8/site-packages/earthdiagnostics/utils.py", line 92, in get_mask
mask_handler = Utils.open_cdf("mask.nc")
File "/opt/conda/lib/python3.8/site-packages/earthdiagnostics/utils.py", line 630, in open_cdf
return netCDF4.Dataset(filepath, mode)
File "src/netCDF4/_netCDF4.pyx", line 2330, in netCDF4._netCDF4.Dataset.__init__
File "src/netCDF4/_netCDF4.pyx", line 1948, in netCDF4._netCDF4._ensure_nc_success
~~~
and in the .err file (`/esarchive/autosubmit/a4a0/tmp/LOG_a4a0/a4a0_19800101_fc00_3_EARTHDIAGS.20220215152227.err`):
~~~
+ singularity exec /esarchive/software/containers/earthdiagnostics/earthdiagnostics-latest.sif earthdiags -f /gpfs/scratch/bsc32/bsc32283/a4a0/diags_a4a0_19800101_fc00_3.conf -lc DEBUG
^[[33mWARNING:^[[0m seccomp requested but not enabled, seccomp library is missing or too old
HDF5-DIAG: Error detected in HDF5 (1.10.6) thread 47495698716416:
#000: H5F.c line 509 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1535 in H5F_open(): file is already open for read-only
major: File accessibilty
minor: Unable to open file
~~~
I'm not sure what is this `mask.nc` and I don't understand why it's failing for this chunk.
Could you help me here?
Thankssloosvelsloosvelhttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/141Issue with interpolation with interpcdo2021-10-28T16:55:52+02:00Valentina SicardiIssue with interpolation with interpcdoHi,
in the interpolation with earthdiagnostics we (my self and other collegues) are facing some problems. This problem may be due to the fact that the data on orca grid " have an extra column on the left and right side. These contain a ...Hi,
in the interpolation with earthdiagnostics we (my self and other collegues) are facing some problems. This problem may be due to the fact that the data on orca grid " have an extra column on the left and right side. These contain a replication of the second and second last column. This hides the cyclic interpolation over the edges. A similar problem concerns the first and last row. With selindexbox these columns and rows can be removed" (cit from cdo forum) and it is a known problem for some models: https://code.mpimet.mpg.de/boards/1/topics/8676.
But we never changed the orca grid, so how comes that in the past we did not have this problem?
We discussed about it with @rfernand and realized that it is probably due to the CDO version used on nord3.
On nord3 there is v1.9.8, while on the WSs the default cdo version is 1.6.3.
Here an exammple of interpolation done with earthdiagnostics:
![earthd_sst](/uploads/6e476d6bfd0da5b11e9a554de6abbbd8/earthd_sst.png)
and here an example of interpolated done offline on nord3 with CDO, with the trim of the extra orca band:
![sst_correcty_interp](/uploads/4e6922ed40f8c1224d61fc2e86c82d9b/sst_correcty_interp.png)
Note that these two plots are an example, they are from 2 different years and runs.
It is quite important we fix it.
thank you
@jvegas
FYI: @acarreri @portega @jacosta @vlapin @mgalitap @jllort @eexarchou
This issue is related to this: https://earth.bsc.es/gitlab/es/requests/-/issues/1563 and https://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/139https://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/140siasiesiv diagnostic: failed to merge into a single cube2021-08-17T13:09:23+02:00acarrerisiasiesiv diagnostic: failed to merge into a single cubeHi @jvegas @sloosvel ,
I have an error message when using Earthdiagnostic on HR experiment (cf for instance this log file: `/esarchive/scratch/Earth/acarreri/earthdiags_suite/suite_home/a3qa_diagnostics/19870101/Member_0/Chunks_1/run.1`...Hi @jvegas @sloosvel ,
I have an error message when using Earthdiagnostic on HR experiment (cf for instance this log file: `/esarchive/scratch/Earth/acarreri/earthdiags_suite/suite_home/a3qa_diagnostics/19870101/Member_0/Chunks_1/run.1`):
~~~
^[[31m[ERROR] Siasiesiv Startdate: 19870101 Member: 0 Chunk: 1 Basins: Barkara,Labrador_Sea,Labrador_Sea_Nudging,Labrador_Sea_conv,Labrador_Sea_conv2,Northern_Hemisphere Omit volume: False: failed to merge into a single cube.
cube.cell_methods differ
File "/opt/conda/lib/python3.7/site-packages/earthdiagnostics/work_manager.py", line 313, in _run_job
job.compute()
File "/opt/conda/lib/python3.7/site-packages/earthdiagnostics/ocean/siasiesiv.py", line 218, in compute
self.save()
File "/opt/conda/lib/python3.7/site-packages/earthdiagnostics/ocean/siasiesiv.py", line 260, in save
self.generated[var].set_local_file(temp, diagnostic=self)
File "/opt/conda/lib/python3.7/site-packages/earthdiagnostics/datafile.py", line 318, in set_local_file
self.prepare_to_upload(rename_var)
File "/opt/conda/lib/python3.7/site-packages/earthdiagnostics/datafile.py", line 276, in prepare_to_upload
self._prepare_region()
File "/opt/conda/lib/python3.7/site-packages/earthdiagnostics/datafile.py", line 478, in _prepare_region
final_cube = cube_list.merge_cube()
File "/opt/conda/lib/python3.7/site-packages/iris/cube.py", line 396, in merge_cube
proto_cube.register(cube, error_on_mismatch=True)
File "/opt/conda/lib/python3.7/site-packages/iris/_merge.py", line 1275, in register
match = cube_signature.match(other, error_on_mismatch)
File "/opt/conda/lib/python3.7/site-packages/iris/_merge.py", line 411, in match
raise iris.exceptions.MergeError(msgs)
^[[0m^[[39m
^[[31m[ERROR] Total wasted time: 0:00:06.702166^[[0m^[[39m
~~~
It could come from the fact that because on this experiment, the `EARTHDIAGNOSTICS` online tool was activated and a file already existed before running this diagnostic with earthdiagnotics suite offline. (even if I think that for other variables, it was also the case, with existing files, but I got an error only for this variable).
Another thing is that the final files seem correct, with the new region asked included. So at the end, the diagnostic seem to run correctly, despite this error message.
It seems the same kind of error that I got with the "moc" variable (cf this [issue](https://earth.bsc.es/gitlab/es/earthdiags_suite/-/issues/6)), even if the error message is not the same.
Can I trust the output results despite the error message?
Could you in that case remove this kind of "bug"?
Many thanks in advance.
AudeJavier Vegas-Regidorjavier.vegas@bsc.esJavier Vegas-Regidorjavier.vegas@bsc.eshttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/139Issue with conservative interpolation2021-10-08T10:53:05+02:00Valentina SicardiIssue with conservative interpolationHi @jvegas
I tried the conservative interpolation, but I get this error:
```
STDERR:cdo remap: YAC first order conservative weights from curvilinear (362x292) to lonlat (360x180) grid, with source mask (65544)
ERROR: invalid cell
...Hi @jvegas
I tried the conservative interpolation, but I get this error:
```
STDERR:cdo remap: YAC first order conservative weights from curvilinear (362x292) to lonlat (360x180) grid, with source mask (65544)
ERROR: invalid cell
Aborting in file clipping.c, line 1295 ...
^[[31m[ERROR] Job Interpolate with CDO Startdate: 19800101 Member: 0 Chunk: 1 Variable: ocean:fgco2 Frequency: mon Target grid: r360x180 Original grid: Mask ocean: True Model: Ec3.2_O1L75 failed (<class 'cdo.CDOException'>): (returncode:1) cdo remap: YAC first order conservative weights from curvilinear (362x292) to lonlat (360x180) grid, with source mask (65544)
ERROR: invalid cell
```
Can you please have a look to see if it is easy to fix?
Logs here:
/esarchive/scratch/Earth/vsicardi/post_proc/earthdiags_suite/suite_home/a3qm_dev_diagnostics/19800101/Member_0/Chunks_1
Thank youhttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/138Error due to wrong path- Earthiagnostic online2021-06-17T18:04:59+02:00Valentina SicardiError due to wrong path- Earthiagnostic onlineHi,
I am running an NEMO-PISCES exp in which I define a specific outclass and I would like to run earthiagnostics online. I define in proj.conf the diagnostics I want.
The job EARTHDIAGS fails becasue is going to search for the input i...Hi,
I am running an NEMO-PISCES exp in which I define a specific outclass and I would like to run earthiagnostics online. I define in proj.conf the diagnostics I want.
The job EARTHDIAGS fails becasue is going to search for the input in the wrong path, this is the error message:
`FileNotFoundError: [Errno 2] No such file or directory: '/esarchive/exp/ecearth/a3qm/cmorfiles/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Oyear/talk/gn'`
If I use another outclass and add to BASIC_PISCES some diagnostics I want, I do not get this error.
What am I doing wrong?
The exp is **a3qm**
@jvegas can you please have a look?
FYI: @rbernard @portegahttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/137Issue in the integration of the OHC in HR experiment2022-07-21T11:28:42+02:00acarreriIssue in the integration of the OHC in HR experimentHi @jvegas , @sloosvel
There is an issue in the computation of the ocean heat content when regionally integrated for HR experiments. For the spatial maps of OHC integrated between 0 and 300m for instance, it seems to be correct. But th...Hi @jvegas , @sloosvel
There is an issue in the computation of the ocean heat content when regionally integrated for HR experiments. For the spatial maps of OHC integrated between 0 and 300m for instance, it seems to be correct. But the total ocean heat content are empty, whatever the region asked.
It's probably the same issue than mentioned [here](https://earth.bsc.es/gitlab/es/auto-ecearth3/-/issues/1329#note_136406) for the heat content part.
Can you have a look at it?
Thanks
Audeacarreriacarrerihttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/136problem with interpcdo density2021-04-09T20:30:54+02:00Valentina Sicardiproblem with interpcdo densityHi @jvegas
for some corrections we had to do to the official omips runs, I am redoing most of the diagnostics for the new formatted omips runs.
I try to interpolate to regular grid the diagnostics density.
The calculation of the den...Hi @jvegas
for some corrections we had to do to the official omips runs, I am redoing most of the diagnostics for the new formatted omips runs.
I try to interpolate to regular grid the diagnostics density.
The calculation of the density goes smooth, nevertheless the interpolation fails.
I noticed that the generated variable is: `float sigma0(time, lev, dim2, dim3) ` as you can check here:
`/esarchive/exp/ecearth/omip1/diags/OMIP/EC-Earth-Consortium/EC-Earth3/omip1/r1i1p1f1/Omon/sigma0/gn/v20200310`
while in the past for the old omip, it was: `float sigma0(time, lev, j, i) ;`, as you can check here:
`/esarchive/exp/ecearth/a2l5_sixth/diags/OMIP/EC-Earth-Consortium/EC-Earth3/omip1/r1i1p1f1/Omon/sigma0/gn/v20200316`
the error at the interpolation level can be found here:
`/esarchive/scratch/Earth/vsicardi/temporal_earth/earthdiags_suite/suite_home/omip1_diagnostics/19480101/Member_0/Chunks_1`
thank youJavier Vegas-Regidorjavier.vegas@bsc.esJavier Vegas-Regidorjavier.vegas@bsc.eshttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/135Issue in the AMOC computation for HR experiment2022-09-27T12:10:49+02:00acarreriIssue in the AMOC computation for HR experimentHi @jvegas,
First of all, Happy New Year!
I've got some troubles with the AMOC computation on HR experiments. The outputs have null values north of around 65°N, cf the figure:
![AMOC_streamfunction_a2s5_clim_50ans_2348-2397_HR_tuning_e...Hi @jvegas,
First of all, Happy New Year!
I've got some troubles with the AMOC computation on HR experiments. The outputs have null values north of around 65°N, cf the figure:
![AMOC_streamfunction_a2s5_clim_50ans_2348-2397_HR_tuning_exp](/uploads/8f2824dab282b1f50025d6d3a632046d/AMOC_streamfunction_a2s5_clim_50ans_2348-2397_HR_tuning_exp.png)
You can find the related files there: `/esarchive/exp/ecearth/a2s5/diags/CMIP/EC-Earth-Consortium/EC-Earth3-HR/piControl/r1i1p1f1/Omon/vsftmyz/gn/v20200425/`
It could be related to a wrong mas applied? Could you have a look at it please?
Thanks.
AudeJavier Vegas-Regidorjavier.vegas@bsc.esJavier Vegas-Regidorjavier.vegas@bsc.eshttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/134Issue with the member names built from create_suite.py2021-01-15T12:50:00+01:00Danila VolpiIssue with the member names built from create_suite.pyHi @jvegas and @sloosvel ,
I am trying to use earthdiags_suite for a DCPP experiment that has been downloaded from ESGF from @msamso on /esarchive and I have a couple of issues. The first one is that the original name of the experiment ...Hi @jvegas and @sloosvel ,
I am trying to use earthdiags_suite for a DCPP experiment that has been downloaded from ESGF from @msamso on /esarchive and I have a couple of issues. The first one is that the original name of the experiment is "cmip6-dcppA-hindcast_i2p1", but earthdiags does not accept "-" in the name of the experiment, so I have created a symbolic link in the folder called "cmip6_dcppA_hindcast_i2p1". It is an acceptable workaround, but it can be confusing for other users that need the same dataset having very similar names of the same experiment (and would be a pity for me to create a name which I'm the only one to understand). So I was wondering if there is any chance that earthdiags could work with the original name, which contains the character "-".
However, the real reason for which I cannot run the diagnostics is that the members of the experiment are called r6i**2**p1f1- r7i**2**p1f1 and so on, with a **2** in the name, instead of the usual **1**. Is there a way to force earthdiags to accept the name of the members, without using its default?
Thank you very much in advance for your help!https://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/133Error with density diag interpolation2020-12-10T15:34:07+01:00Valentina SicardiError with density diag interpolationHi @jvegas,
I try to produce density intepolated to regular grid.
I define it in this way:
suite.diags = 'density interpcdo,ocean,sigma0,r360x180'
> Interpolate with CDO Startdate: 19580101 Member: 0 Chunk: 1 Variable: ocean:sigm...Hi @jvegas,
I try to produce density intepolated to regular grid.
I define it in this way:
suite.diags = 'density interpcdo,ocean,sigma0,r360x180'
> Interpolate with CDO Startdate: 19580101 Member: 0 Chunk: 1 Variable: ocean:sigma0 Frequency: mon Target grid: r360x180 Original grid: Mask ocean: True Model: Ec3.2_O1L75: (returncode:1) Warning (cdfInqContents): Coordinates variable time can't be assigned!
> Error (cdf_put_vara_double): NetCDF: Numeric conversion not representable
the logs and configuration files are here:
/esarchive/scratch/Earth/vsicardi/earthds_nord3/earthdiags_suite/create_suite_a2er_sixth.py
/esarchive/scratch/Earth/vsicardi/earthds_nord3/earthdiags_suite/suite_home/a2er_sixth_diagnostic
Please, when fix it, if possible take into account this other issue: https://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/132
any help is really appreciateJavier Vegas-Regidorjavier.vegas@bsc.esJavier Vegas-Regidorjavier.vegas@bsc.eshttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/132ncview can not show density output2020-12-10T12:01:50+01:00Javier Vegas-Regidorjavier.vegas@bsc.esncview can not show density outputHi @jvegas,
I have been able to compute the density diagnostic without issues, nevertheless I would like to ask if it was possible to add a mask to the final output.
I added a mask with cdo or python, but it would be nice to have the f...Hi @jvegas,
I have been able to compute the density diagnostic without issues, nevertheless I would like to ask if it was possible to add a mask to the final output.
I added a mask with cdo or python, but it would be nice to have the final diagnostic already masked.
You can see an example of original output here:
/esarchive/exp/ecearth/a2er_sixth/diags/OMIP/EC-Earth-Consortium/EC-Earth3/omip2/r1i1p1f1/Omon/sigma0/gn/v20200301/
you can check that ncview gives this error:
```
Note: no Ncview app-defaults file found, using internal defaults
calculating min and maxes for sigma0...
*** genlevs: step =-1, failed
*** genlevs: Input data: mindat=-inf maxdat=1029.308105 nlevels=9
*** trialstep=inf, mant=inf, exp=-2147483648.000000
```
after masking it, there is no error.
Here you can find a masked file I created as test:
/esarchive/scratch/Earth/vsicardi/OMIP/a2er_sixth/test_mask.nc
thank youJavier Vegas-Regidorjavier.vegas@bsc.esJavier Vegas-Regidorjavier.vegas@bsc.eshttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/131Region mask and how to use their name in the netcdf file2020-09-03T11:37:04+02:00acarreriRegion mask and how to use their name in the netcdf fileHi @jvegas, @sloosvel,
Are the files used to define the different regions that we can call in EarthDiagnostics those:
/esarchive/autosubmit/basins.Ec3.2_O1L75.nc and basins.Ec3.2_O25L75.nc ?
It's mentioned on this [page](https://earth.b...Hi @jvegas, @sloosvel,
Are the files used to define the different regions that we can call in EarthDiagnostics those:
/esarchive/autosubmit/basins.Ec3.2_O1L75.nc and basins.Ec3.2_O25L75.nc ?
It's mentioned on this [page](https://earth.bsc.es/gitlab/es/earthdiags_suite/-/wikis/Suite-configuration) of the wiki of the suite, but because things are moving fast, I prefer to ask for confirmation.
If these are the files, how can I use the names of the region? I mean, they seem to be defined in a precise format, with the same number of characters (with a lot of spaces after the name). I'd like to put in argument of my script the name of one region and select it in the netcdf with xarray, but I don't know how to convert my string to be consistent with the one on the netcdf. I don't talk about the utf-8 format, but about the length of the name.
Not sure I'm clear?! But thanks in advance.
AudeJavier Vegas-Regidorjavier.vegas@bsc.esJavier Vegas-Regidorjavier.vegas@bsc.eshttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/130Error with yearly data2020-08-18T19:04:25+02:00Valentina SicardiError with yearly dataHi @jvegas and @sloosvel
How do I run a suite with yearly output? Can I do at the same time with monthly, or should I launch the suite twice?
This is important for some obgc variable such as o2 and no3, that we have generally yearly....Hi @jvegas and @sloosvel
How do I run a suite with yearly output? Can I do at the same time with monthly, or should I launch the suite twice?
This is important for some obgc variable such as o2 and no3, that we have generally yearly. Can you suggest me what to set in the create_suite.py, please?
I remember we had a similar issue with the online basic pisces diagnostic, but it was solved, isn't it?
thanksJavier Vegas-Regidorjavier.vegas@bsc.esJavier Vegas-Regidorjavier.vegas@bsc.eshttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/129zonalmean outputs NaNs2020-09-10T16:11:38+02:00sloosvelzonalmean outputs NaNsDue to out of bounds error. Adding merge request in diagonals.Due to out of bounds error. Adding merge request in diagonals.sloosvelsloosvelhttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/127No space left on device2020-07-27T12:01:26+02:00aamaralarthur.amaral@bsc.esNo space left on deviceHi,
I'm running a piControl experiment (a2y7) and had a `No space left on device` in of my EARTHDIAGS jobs (chunk 11):
```
singularity exec ${CONTAINER} earthdiags -f ${conf_file} -lc DEBUG
+ singularity exec /esarchive/software/contai...Hi,
I'm running a piControl experiment (a2y7) and had a `No space left on device` in of my EARTHDIAGS jobs (chunk 11):
```
singularity exec ${CONTAINER} earthdiags -f ${conf_file} -lc DEBUG
+ singularity exec /esarchive/software/containers/earthdiagnostics/earthdiagnostics-3.4.0.sif earthdiags -f /gpfs/scratch/bsc32/bsc32436/a2y7/diags_a2y7_18500101_fc0_11.conf -lc DEBUG
^[[33mWARNING:^[[0m seccomp requested but not enabled, seccomp library is missing or too old
Traceback (most recent call last):
File "/opt/conda/bin/earthdiags", line 23, in <module>
main()
File "/opt/conda/bin/earthdiags", line 17, in main
if not EarthDiags.parse_args(sys.argv[1:]):
File "/opt/conda/lib/python3.8/site-packages/earthdiagnostics/earthdiags.py", line 178, in parse_args
result = diags.run()
File "/opt/conda/lib/python3.8/site-packages/earthdiagnostics/earthdiags.py", line 229, in run
self._prepare_mesh_files()
File "/opt/conda/lib/python3.8/site-packages/earthdiagnostics/earthdiags.py", line 438, in _prepare_mesh_files
self._prepare_mesh_using_scratch(
File "/opt/conda/lib/python3.8/site-packages/earthdiagnostics/earthdiags.py", line 473, in _prepare_mesh_using_scratch
if self._copy_file(
File "/opt/conda/lib/python3.8/site-packages/earthdiagnostics/earthdiags.py", line 518, in _copy_file
shutil.copyfile(source, destiny)
File "/opt/conda/lib/python3.8/shutil.py", line 272, in copyfile
_fastcopy_sendfile(fsrc, fdst)
File "/opt/conda/lib/python3.8/shutil.py", line 163, in _fastcopy_sendfile
raise err from None
File "/opt/conda/lib/python3.8/shutil.py", line 149, in _fastcopy_sendfile
sent = os.sendfile(outfd, infd, offset, blocksize)
OSError: [Errno 28] No space left on device: '/esarchive/autosubmit/con_files/mesh_mask_nemo.Ec3.2_O1L75.nc' -> 'cmip6/mesh_mask_nemo.Ec3.2_O1L75.nc'
SUCCESS=${?}
+ SUCCESS=1
rm -r ${TMPDIR}
```
Here's the log: `/esnas/autosubmit/a2y7/tmp/LOG_a2y7/a2y7_18500101_fc0_11_EARTHDIAGS.20200626221630.err`
@jvegas @etourign @rbernardJavier Vegas-Regidorjavier.vegas@bsc.esJavier Vegas-Regidorjavier.vegas@bsc.eshttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/126ohc 0-1002021-04-16T16:07:19+02:00Eleftheria Exarchouohc 0-100Hi @jvegas
I independently calculated ohc 0-100 m for 1 file in a2i9 (here `/esarchive/scratch/eexarcho/Eleftheria/PRIMAVERA/WP2/ORAS4_ohc/calc_ohc0-100-test-a2i9.sh`) and compared it to the earthdiagnostics output
`/esarchive/exp/ec...Hi @jvegas
I independently calculated ohc 0-100 m for 1 file in a2i9 (here `/esarchive/scratch/eexarcho/Eleftheria/PRIMAVERA/WP2/ORAS4_ohc/calc_ohc0-100-test-a2i9.sh`) and compared it to the earthdiagnostics output
`/esarchive/exp/ecearth/a2i9/diags/DCPP/EC-Earth-Consortium/EC-Earth3/dcppA-hindcast/r1i1p1f1/Omon/heatc0-100m/r360x180/v20200122`
and I got different results
![image](/uploads/0a7b47f1073cd66a01ed486220faa87c/image.png)
I am wondering what might have been wrong...
my calculation is
```
tmpdir=/esarchive/scratch/eexarcho/Eleftheria/PRIMAVERA/WP2/ORAS4_ohc/meshmask/
tmask=${tmpdir}/tmask.nc
tmask1=${tmpdir}/tmask1.nc
indir=/esarchive/exp/ecearth/a2i9/cmorfiles/DCPP/EC-Earth-Consortium/EC-Earth3/dcppA-hindcast/r1i1p1f1/Omon/thetao/gn/v20200122/
infile=${indir}/thetao_Omon_EC-Earth3_dcppA-hindcast_s1980-r1i1p1f1_gn_198006-198101.nc
fileout=ohc0-100_Omon_EC-Earth3_dcppA-hindcast_s1980-r1i1p1f1_gn_198006-198101.nc
cdo selvar,thetao $infile tmp1.nc
cdo ifthen $tmask tmp1.nc tmp2.nc
cdo -addc,273.15 tmp2.nc tmp3.nc
ncrename -O -v thetao,ohc tmp3.nc
cdo mul tmp3.nc ${tmpdir}/e3tfile.nc tmp4.nc
cdo setzaxis,${tmpdir}/lev.asc tmp4.nc tmp5.nc
ncks -O -d depth,0.0,100.0 tmp5.nc tmp6.nc
cdo mulc,4082793 tmp6.nc tmp7.nc
cdo vertsum tmp7.nc tmp8.nc
cdo ifthen ${tmpdir}/tmask1.nc tmp8.nc $fileout
var=ohc
units='J/m2'
long_name='ocean heat content integrated between 0-100 m'
# long name
ncatted -O -a long_name,${var},c,c,"${long_name}" $fileout
# units
ncatted -O -a units,${var},c,c,"${units}" $fileout
```
which is pretty simple. Perhaps we use different volumetric heat capacity cp (cp=4082793 above)?https://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/125regmean, siesiasiv and moc diags Error2021-04-16T10:56:30+02:00flope1regmean, siesiasiv and moc diags ErrorHello @jvegas
I've found a recurrent error when I try to run siesiasiv and moc diagnostics in some experiments (in other experiments with a similar configuration it goes fine...). Even the error showing the output of these diagnostics...Hello @jvegas
I've found a recurrent error when I try to run siesiasiv and moc diagnostics in some experiments (in other experiments with a similar configuration it goes fine...). Even the error showing the output of these diagnostics seems to be fine and well computed.
Here ther's an example of the recurrent error with siesiasiv: `/esarchive/scratch/Earth/flope1/earthdiags_suite/suite_home/a2er_sixth_diagnostics/19800101/Member_0/Chunks_37/run_custom.1`
and here with moc:
`/esarchive/scratch/Earth/flope1/earthdiags_suite/suite_home/a2w0_diagnostics/19800201/Member_1/run_custom.1`
Thanks,
Ferran LópezJavier Vegas-Regidorjavier.vegas@bsc.esJavier Vegas-Regidorjavier.vegas@bsc.es