Earth Diagnostics issueshttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues2021-10-28T16:55:52+02:00https://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/141Issue with interpolation with interpcdo2021-10-28T16:55:52+02:00Valentina SicardiIssue with interpolation with interpcdoHi,
in the interpolation with earthdiagnostics we (my self and other collegues) are facing some problems. This problem may be due to the fact that the data on orca grid " have an extra column on the left and right side. These contain a ...Hi,
in the interpolation with earthdiagnostics we (my self and other collegues) are facing some problems. This problem may be due to the fact that the data on orca grid " have an extra column on the left and right side. These contain a replication of the second and second last column. This hides the cyclic interpolation over the edges. A similar problem concerns the first and last row. With selindexbox these columns and rows can be removed" (cit from cdo forum) and it is a known problem for some models: https://code.mpimet.mpg.de/boards/1/topics/8676.
But we never changed the orca grid, so how comes that in the past we did not have this problem?
We discussed about it with @rfernand and realized that it is probably due to the CDO version used on nord3.
On nord3 there is v1.9.8, while on the WSs the default cdo version is 1.6.3.
Here an exammple of interpolation done with earthdiagnostics:
![earthd_sst](/uploads/6e476d6bfd0da5b11e9a554de6abbbd8/earthd_sst.png)
and here an example of interpolated done offline on nord3 with CDO, with the trim of the extra orca band:
![sst_correcty_interp](/uploads/4e6922ed40f8c1224d61fc2e86c82d9b/sst_correcty_interp.png)
Note that these two plots are an example, they are from 2 different years and runs.
It is quite important we fix it.
thank you
@jvegas
FYI: @acarreri @portega @jacosta @vlapin @mgalitap @jllort @eexarchou
This issue is related to this: https://earth.bsc.es/gitlab/es/requests/-/issues/1563 and https://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/139https://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/145Order of dimensions changed for heatsum2024-01-22T10:38:15+01:00Pierre-Antoine BretonnièreOrder of dimensions changed for heatsum2 files generated with the same config file but for different dates have a different order of dimension, preventing them from properly concatenating in the monitor job.
Discovered in t0e7:
```
pbretonn@bscearth319:/esarchive/exp/eceart...2 files generated with the same config file but for different dates have a different order of dimension, preventing them from properly concatenating in the monitor job.
Discovered in t0e7:
```
pbretonn@bscearth319:/esarchive/exp/ecearth/t0e7/diags/DCPP/EC-Earth-Consortium/EC-Earth3/dcppA-hindcast/r9i1p1f1$ ncdump -h ./Omon/heatcsum0-700m/gn/v20240116/heatcsum0-700m_Omon_EC-Earth3_dcppA-hindcast_s2001-r9i1p1f1_gn_200101-200112.nc
netcdf heatcsum0-700m_Omon_EC-Earth3_dcppA-hindcast_s2001-r9i1p1f1_gn_200101-200112 {
dimensions:
region = 5 ;
time = 12 ;
region_length = 15 ;
variables:
double time(time) ;
time:axis = "T" ;
time:units = "days since 1850-01-01 00:00:00" ;
time:standard_name = "time" ;
time:long_name = "time" ;
time:calendar = "proleptic_gregorian" ;
double heatcsum0-700m(**region, time**) ;
heatcsum0-700m:long_name = "Total Ocean heat content" ;
heatcsum0-700m:units = "J" ;
heatcsum0-700m:coordinates = "region time" ;
heatcsum0-700m:standard_name = "total_ocean_heat_content" ;
heatcsum0-700m:_FillValue = 1.e+20 ;
heatcsum0-700m:missingValue = 1.e+20 ;
char region(region, region_length) ;
region:standard_name = "region" ;
// global attributes:
:invalid_standard_name = "total_ocean_heat_content" ;
:missingValue = 1.e+20 ;
:modeling_realm = "ocean" ;
:table_id = "Table Omon (December 2013)" ;
:Conventions = "CF-1.7" ;
:NCO = "netCDF Operators version 4.9.9 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
:history = "UTC 2024-01-18T07:30:49.514896: Diagnostic Heat content layer Startdate: 20010101 Member: 9 Chunk: 1 Box: 0-700m calculated with EarthDiagnostics version 3.5.9;UTC 2024-01-18T07:30:49.512999: Using Basins masks file version with grid Ec3.2_O1L75. Original file can be found in /esarchive/autosubmit/conf_files.;Thu Jan 18 08:30:49 2024: /gpfs/projects/bsc32/software/suselinux/11/software/earthdiagnostics/3.5.9/bin/ncatted -O -a _FillValue,heatcsum0-700m,o,d,1.e20 -a missingValue,heatcsum0-700m,o,d,1.e20 --nco_dbg_lvl=3 --output=/scratch/tmp/2730916/diags/t0e7/templuj2p2cc.nc /scratch/tmp/2730916/diags/t0e7/templuj2p2cc.nc\nThu Jan 18 08:30:49 2024: /gpfs/projects/bsc32/software/suselinux/11/software/earthdiagnostics/3.5.9/bin/ncatted -O -a _FillValue,heatcsum0-700m,o,d,1.e20 -a missingValue,heatcsum0-700m,o,d,1.e20 --nco_dbg_lvl=3 --output=/scratch/tmp/2730916/diags/t0e7/templuj2p2cc.nc /scratch/tmp/2730916/diags/t0e7/templuj2p2cc.nc" ;
:_NCProperties = "version=2,netcdf=4.7.4,hdf5=1.10.6" ;
}
(base) pbretonn@bscearth319:/esarchive/exp/ecearth/t0e7/diags/DCPP/EC-Earth-Consortium/EC-Earth3/dcppA-hindcast/r9i1p1f1$ ncdump -h ./Omon/heatcsum0-700m/gn/v20240116/heatcsum0-700m_Omon_EC-Earth3_dcppA-hindcast_s2001-r9i1p1f1_gn_200201-200212.nc
netcdf heatcsum0-700m_Omon_EC-Earth3_dcppA-hindcast_s2001-r9i1p1f1_gn_200201-200212 {
dimensions:
time = 12 ;
region = 5 ;
region_length = 50 ;
variables:
double time(time) ;
time:bounds = "time_bnds" ;
time:units = "days since 1850-01-01 00:00:00" ;
time:calendar = "proleptic_gregorian" ;
time:axis = "T" ;
time:long_name = "time" ;
time:standard_name = "time" ;
char region(region, region_length) ;
double heatcsum0-700m(**time, region**) ;
heatcsum0-700m:units = "J" ;
heatcsum0-700m:coordinates = "region time" ;
heatcsum0-700m:standard_name = "total_ocean_heat_content" ;
heatcsum0-700m:long_name = "Total Ocean heat content" ;
heatcsum0-700m:_FillValue = 1.e+20 ;
heatcsum0-700m:missingValue = 1.e+20 ;
// global attributes:
:modeling_realm = "ocean" ;
:table_id = "Table Omon (December 2013)" ;
:NCO = "netCDF Operators version 4.9.9 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
:history = "UTC 2024-01-18T07:16:40.625440: Diagnostic Heat content layer Startdate: 20010101 Member: 9 Chunk: 2 Box: 0-700m calculated with EarthDiagnostics version 3.5.9;UTC 2024-01-18T07:16:40.623510: Using Basins masks file version with grid Ec3.2_O1L75. Original file can be found in /esarchive/autosubmit/conf_files.;Thu Jan 18 08:16:40 2024: /gpfs/projects/bsc32/software/suselinux/11/software/earthdiagnostics/3.5.9/bin/ncatted -O -a _FillValue,heatcsum0-700m,o,d,1.e20 -a missingValue,heatcsum0-700m,o,d,1.e20 --nco_dbg_lvl=3 --output=/scratch/tmp/2730895/diags/t0e7/tempw6g88lus.nc /scratch/tmp/2730895/diags/t0e7/tempw6g88lus.nc" ;
:_NCProperties = "version=2,netcdf=4.7.4,hdf5=1.10.6" ;
```
Rerunning the job fixes it (to be confirmed by @eferre1 ) and the job is anyway being replaced by ESMValTool by @sghosh but commenting it here for proper documentationhttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/144Python version update in docs?2022-09-07T18:15:58+02:00Eneko Martineneko.martin@bsc.esPython version update in docs?As I can see in [environment.yml](https://earth.bsc.es/gitlab/es/earthdiagnostics/-/blob/master/environment.yml) Python 3.x is supported (it must be lower than 3.9). However, the documentation says that it only works with Python 2.x ([se...As I can see in [environment.yml](https://earth.bsc.es/gitlab/es/earthdiagnostics/-/blob/master/environment.yml) Python 3.x is supported (it must be lower than 3.9). However, the documentation says that it only works with Python 2.x ([see installation requirements](https://earthdiagnostics.readthedocs.io/en/stable/tutorial.html#installation)). Then, I suppose that the required Python version should be updated in docs.https://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/143Homogenization of basins files used by Earthdiagnostics2023-04-27T15:05:35+02:00acarreriHomogenization of basins files used by EarthdiagnosticsHi Earthdiagnostics potential users, @vlapin, @rfernand, @vsicardi , @jacosta , @yruprich , @eexarchou , @emoreno, @etourign
hi @mcastril , @eferre1,
Continuing in the line of trying to homogenize our diagnoses, I raise this issue abo...Hi Earthdiagnostics potential users, @vlapin, @rfernand, @vsicardi , @jacosta , @yruprich , @eexarchou , @emoreno, @etourign
hi @mcastril , @eferre1,
Continuing in the line of trying to homogenize our diagnoses, I raise this issue about the basin masks used in Earthdiagnostics, which is now implemented in the workflow of auto-ecearth.
As you may know, you can ask for diagnostics (`regmean` but also `moc`) to be computed in specific regions.
Earthdiagnostics uses for that the `basins.Ec${version}_${RES}.nc` in `/esarchive/autosubmit/con_files/`.
Right now, there is:
- basins.Ec3.2_O1L75.nc
- basins.Ec3.2_O25L75.nc
- basins.Ec3.6_12L75.nc (that @emoreno recently produced).
**BUT** the different regions are not consistent between grids!
For instance, the first region, named `AMV_North_Atlantic` looks like, depending on the grid (from ORCA1 to ORCA012): ![Screenshot_from_2022-02-23_13-39-06](/uploads/052be7f342ecb3eed06d9347c1fe0ca3/Screenshot_from_2022-02-23_13-39-06.png)
These `basins` files are produced by this [script](https://earth.bsc.es/gitlab/cp/cpg-shared-tools/-/blob/master/New%20Diagnostics/Climate%20Indices/Basin%20and%20Regional%20Masks/create_basins_mask.py) and uses the [basins.yml](https://earth.bsc.es/gitlab/cp/cpg-shared-tools/-/tree/master/New%20Diagnostics/Climate%20Indices/Basin%20and%20Regional%20Masks) for the definition of the regions.
Three things here:
- the regions should be consistent between grids except that this `basins.yml` file uses for the main oceans (actually only `Atlantic_Ocean` is called in the `basins.yml` file) the basins from Nemo directly, that is from this file: `/esarchive/autosubmit/con_files/new_maskglo.Ec3.[Screenshot_from_2022-02-23_14-56-20](/uploads/be909a85f752ac2f602bb4ffc06307c9/Screenshot_from_2022-02-23_14-56-20.png)
- some regions such as `AMV_North_Atlantic` may have been created in a different way for ORCA1 and ORCA025, that's why it's different from the one in ORCA12 (which uses the correct definition in `basins.yml`).
- some regions already included in ORCA1 and ORCA025 are not present in the new ORCA12 file (`basins.Ec3.6_12L75.nc`), because these regions haven't been updated in our `basins.yml` in cpg_shared_tool project. This file shows the different regions present or not: [Earthdiagnostics_regions.ods](/uploads/f311c87904d4be3dcf37b45f667e6520/Earthdiagnostics_regions.ods)
In any case, it can be the moment to redefine/clean the regions we want. Also because for instance, when calling for the 'moc' function in Earthdiagnostics, if you call it for the `North_Atlantic_Ocean` region, you compute it over the North Atlantic + the whole Arctic in ORCA1 and ORCA12, over only the North Atlantic (not the highest latitudes) in ORCA025 when we may want the North_Atlantic + the extension of the Arctic only in the Atlantic part, not including the other Pacific part of the Arctic.
It can be done easily:
- we have to clean or update the basins.yml file
- we have to check the `new_maskglo.{version}_{res}.nc` file for the latest *3.6* version of NEMO and see if they are coherent between resolution (I think the issue is only for ORCA025 here).
- we have to produce the basins.Ec{version}_{res}.nc for this latest 3.6 version with the `create_basins_mask.py`
Who's with me?
PYI @portegaacarreriacarrerihttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/139Issue with conservative interpolation2021-10-08T10:53:05+02:00Valentina SicardiIssue with conservative interpolationHi @jvegas
I tried the conservative interpolation, but I get this error:
```
STDERR:cdo remap: YAC first order conservative weights from curvilinear (362x292) to lonlat (360x180) grid, with source mask (65544)
ERROR: invalid cell
...Hi @jvegas
I tried the conservative interpolation, but I get this error:
```
STDERR:cdo remap: YAC first order conservative weights from curvilinear (362x292) to lonlat (360x180) grid, with source mask (65544)
ERROR: invalid cell
Aborting in file clipping.c, line 1295 ...
^[[31m[ERROR] Job Interpolate with CDO Startdate: 19800101 Member: 0 Chunk: 1 Variable: ocean:fgco2 Frequency: mon Target grid: r360x180 Original grid: Mask ocean: True Model: Ec3.2_O1L75 failed (<class 'cdo.CDOException'>): (returncode:1) cdo remap: YAC first order conservative weights from curvilinear (362x292) to lonlat (360x180) grid, with source mask (65544)
ERROR: invalid cell
```
Can you please have a look to see if it is easy to fix?
Logs here:
/esarchive/scratch/Earth/vsicardi/post_proc/earthdiags_suite/suite_home/a3qm_dev_diagnostics/19800101/Member_0/Chunks_1
Thank youhttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/138Error due to wrong path- Earthiagnostic online2021-06-17T18:04:59+02:00Valentina SicardiError due to wrong path- Earthiagnostic onlineHi,
I am running an NEMO-PISCES exp in which I define a specific outclass and I would like to run earthiagnostics online. I define in proj.conf the diagnostics I want.
The job EARTHDIAGS fails becasue is going to search for the input i...Hi,
I am running an NEMO-PISCES exp in which I define a specific outclass and I would like to run earthiagnostics online. I define in proj.conf the diagnostics I want.
The job EARTHDIAGS fails becasue is going to search for the input in the wrong path, this is the error message:
`FileNotFoundError: [Errno 2] No such file or directory: '/esarchive/exp/ecearth/a3qm/cmorfiles/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Oyear/talk/gn'`
If I use another outclass and add to BASIC_PISCES some diagnostics I want, I do not get this error.
What am I doing wrong?
The exp is **a3qm**
@jvegas can you please have a look?
FYI: @rbernard @portegahttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/119Regsum diagnostic not working2021-10-28T17:04:23+02:00Roberto BilbaoRegsum diagnostic not workingHi @jvegas @sloosvel,
I am trying to compute the regsum for a set of regions for heatcsum0-300m of a1ua, however it does not seem to work. You can find the logs here: /esarchive/scratch/Earth/rfernand/earthdiags_suite/a1ua/suite_home/a1...Hi @jvegas @sloosvel,
I am trying to compute the regsum for a set of regions for heatcsum0-300m of a1ua, however it does not seem to work. You can find the logs here: /esarchive/scratch/Earth/rfernand/earthdiags_suite/a1ua/suite_home/a1ua_diagnostics/19601101/Member_0/Javier Vegas-Regidorjavier.vegas@bsc.esJavier Vegas-Regidorjavier.vegas@bsc.eshttps://earth.bsc.es/gitlab/es/earthdiagnostics/-/issues/117include density in Earth Diagnostics2022-12-16T13:21:59+01:00Raffaele Bernardelloinclude density in Earth Diagnosticsit would be very useful to include the possibility to calculate potential density with respect to different reference pressures from NEMO output.
These variables are useful for diagnosing water masses among many other things. Those comm...it would be very useful to include the possibility to calculate potential density with respect to different reference pressures from NEMO output.
These variables are useful for diagnosing water masses among many other things. Those commonly used are 3:
sigma0 -----> potential density anomaly with reference pressure of 0 dbar (surface)
sigma2 -----> potential density anomaly with reference pressure of 2000 dbar
sigma4 -----> potential density anomaly with reference pressure of 4000 dbar
the first one is actually available in NEMO but because it wasn't included in the CMIP data requests it was not saved for all experiments.
**First possibility**
I have been generating these variables using octave with a set of scripts in matlab code. These are based on the **Thermodynamic Equation of Seawater - 2010 (http://www.teos-10.org/)** which is what NEMO uses internally.
The same scripts are also available in python (apparently):
https://gitmemory.com/TEOS-10
**Second possibility**
To use **CDFTOOLS** however, I haven't been able to generate anything with the default implementation available at BSC. Moreover, I couldn't find a detailed explanation of what's inside these functions. It might be that they are still based on the previous equation of state (EOS-80).
The functions to calculate the three variables above, for TEOS-10 are:
gsw_sigma0
gsw_sigma2
gsw_sigma4
The input to the TEOS-10 functions are conservative temperature (bightetao) and absolute salinity (so)
The functions to calculate the three variables above, for CDFTOOLS are:
cdfsig0
cdfsigi (specifying the reference depth)
the input to the cdftools functions are described generically as "temperature" and "salinity" which leaves me with the doubt if it refers to potential (thetao) or conservative temperature (bigthetao)
@jvegas @sloosvel @vsicardi @portega @fdoblasreyes