siasiesiv produces empty files with ecFlow
Hi @jvegas
I am using the default earthdiagnostics version (earthdiagnostics/3.1.0-foss-2018b-Python-2.7.15) in ecFlow to calculate the siasiesiv daily for experiment a1qw. Although the file is produced (e.g., /esarchive/exp/ecearth/a1qw/cmorfiles/EC-Earth-Consortium/EC-Earth3-LR/historical/S19580201/day/seaIce/siextentn/r5i1p1/v20190201/siextentn_day_EC-Earth3-LR_historical_S19580201_r5i1p1_201402-201501.nc), it is empty and when making ncdump -h it looks a bit different compared to always (for example, I see region_length = 50 in the dimensions, which I never saw). The ecFlow conf file:
[DIAGNOSTICS]
SKIP_DIAGS_DONE = False
SCRATCH_DIR = $NVME1DIR/$USER
SCRATCH_MASKS =
DATA_DIR = /esarchive
CON_FILES = /esarchive/autosubmit/con_files
FREQUENCY = day
DATA_CONVENTION = SPECS
DIAGS = siasiesiv,Baffin_Bay:Barents_Sea:Beaufort_Sea:Bering:CanArch:Central_Arctic:Chukchi_Sea:East_Siberian_Sea:Greenland_Sea:Hudson:Icelandic_Sea:Kara_Sea:Labrador_Sea:Laptev_Sea:Norwegian_Sea:Okhotsk:Irminger_Sea:TotalArc:Grnland:Baffin:Northern_Hemisphere
MAX_CORES = 1
[EXPERIMENT]
INSTITUTE = EC-Earth-Consortium
MODEL = EC-Earth3-LR
MODEL_VERSION = Ec3.2_O1L75
ATMOS_TIMESTEP = 6
OCEAN_TIMESTEP = 6
EXPID = a1qw
NAME = historical
STARTDATES = 19580201
MEMBERS = 4
MEMBER_DIGITS = 2
MEMBER_PREFIX = fc
MEMBER_COUNT_START = 0
CHUNK_SIZE = 12
CHUNKS = 57
CHUNK_LIST = 57
[CMOR]
SKIP_PREPARE = False
FORCE = False
OCEAN_FILES = False
ATMOSPHERE_FILES = False
CHUNKS = 57
VARIABLE_LIST =
VERSION = v20190201
This is the log of the calculation:
Initialising Earth Diagnostics Version 3.1.0
Running diags for experiment a1qw, startdates [u'19580201'], members [4]
Copying mesh files
Copying file mesh_hgr.nc
File mesh_hgr.nc ready
File mesh_zgr.nc ready
File mask.nc ready
Copying file new_maskglo.nc
File new_maskglo.nc ready
Copying file mask_regions.nc
File mask_regions.nc ready
Copying file mask_regions.3d.nc
File mask_regions.3d.nc ready
[32mMesh files ready![0m[39m
Time to prepare: 0:00:07.746303
Checking data for startdate 19580201 member 4
Checking chunk 57...
CMORizing startdate 19580201 member fc04. Starting at 2019-07-17 10:09:42.640886
Skipping ocean cmorization due to configuration
Skipping atmosphere cmorization due to configuration
[32mCMORized startdate 19580201 member fc04! Elapsed time: 0:00:00.012355
[0m[39m
Running diagnostics
Adding siasiesiv,Baffin_Bay:Barents_Sea:Beaufort_Sea:Bering:CanArch:Central_Arctic:Chukchi_Sea:East_Siberian_Sea:Greenland_Sea:Hudson:Icelandic_Sea:Kara_Sea:Labrador_Sea:Laptev_Sea:Norwegian_Sea:Okhotsk:Irminger_Sea:TotalArc:Grnland:Baffin:Northern_Hemisphere to diagnostic list
Starting to compute at 2019-07-17 10:09:51.779436
Using 1 threads
File /esarchive/exp/ecearth/a1qw/cmorfiles/EC-Earth-Consortium/EC-Earth3-LR/historical/S19580201/day/seaIce/sic/r5i1p1/v20190201/sic_day_EC-Earth3-LR_historical_S19580201_r5i1p1_201402-201501.nc ready!
Starting Siasiesiv Startdate: 19580201 Member: 4 Chunk: 57 Basins: Central_Arctic,Chukchi_Sea,Laptev_Sea,Grnland,Hudson,Baffin_Bay,Okhotsk,Greenland_Sea,Icelandic_Sea,Labrador_Sea,Barents_Sea,Bering,Baffin,Northern_Hemisphere,Beaufort_Sea,Norwegian_Sea,TotalArc,CanArch,Irminger_Sea,East_Siberian_Sea,Kara_Sea Omit volume: True
Current status:
===============
Waiting: 0
Ready to run: 0
Running: 1
Completed: 0
Failed: 0
===============
Current status:
===============
Waiting: 0
Ready to run: 0
Running: 1
Completed: 0
Failed: 0
===============
Current status:
===============
Waiting: 0
Ready to run: 0
Running: 1
Completed: 0
Failed: 0
===============
File /esarchive/exp/ecearth/a1qw/cmorfiles/EC-Earth-Consortium/EC-Earth3-LR/historical/S19580201/day/seaIce/siarean/r5i1p1/v20190201/siarean_day_EC-Earth3-LR_historical_S19580201_r5i1p1_201402-201501.nc uploaded!
File /esarchive/exp/ecearth/a1qw/cmorfiles/EC-Earth-Consortium/EC-Earth3-LR/historical/S19580201/day/seaIce/siextents/r5i1p1/v20190201/siextents_day_EC-Earth3-LR_historical_S19580201_r5i1p1_201402-201501.nc uploaded!
File /esarchive/exp/ecearth/a1qw/cmorfiles/EC-Earth-Consortium/EC-Earth3-LR/historical/S19580201/day/seaIce/siareas/r5i1p1/v20190201/siareas_day_EC-Earth3-LR_historical_S19580201_r5i1p1_201402-201501.nc uploaded!
[32mFinished Siasiesiv Startdate: 19580201 Member: 4 Chunk: 57 Basins: Central_Arctic,Chukchi_Sea,Laptev_Sea,Grnland,Hudson,Baffin_Bay,Okhotsk,Greenland_Sea,Icelandic_Sea,Labrador_Sea,Barents_Sea,Bering,Baffin,Northern_Hemisphere,Beaufort_Sea,Norwegian_Sea,TotalArc,CanArch,Irminger_Sea,East_Siberian_Sea,Kara_Sea Omit volume: True[0m[39m
File /esarchive/exp/ecearth/a1qw/cmorfiles/EC-Earth-Consortium/EC-Earth3-LR/historical/S19580201/day/seaIce/siextentn/r5i1p1/v20190201/siextentn_day_EC-Earth3-LR_historical_S19580201_r5i1p1_201402-201501.nc uploaded!
[32mDiagnostics finished at 2019-07-17 10:13:51.489442[0m[39m
[32mElapsed time: 0:03:59.710006
[0m[39m
Time consumed by each diagnostic class
--------------------------------------
siasiesiv 0:03:58.966678
And the run_custom.1.err, which is the only suspicious:
+ set -o pipefail
+ touch /esarchive/scratch/ruben/shock_comparison/processing_data/ec_flow/a1qw_suite/suite_home/a1qw_diagnostics/19580201/Member_4/Chunks_57/run_custom.job1.started
+ trap ERROR 0
+ trap '{ echo "Killed by a signal"; touch /esarchive/scratch/ruben/shock_comparison/processing_data/ec_flow/a1qw_suite/suite_home/a1qw_diagnostics/19580201/Member_4/Chunks_57/run_custom.job1.failed; ERROR ; }' 1 2 3 4 5 6 7 8 10 12 13 15
+ module unuse /apps/modules/modulefiles/applications
++ /apps/modules/LMOD/7.7/lmod/lmod/libexec/lmod bash unuse /apps/modules/modulefiles/applications
+ eval 'MODULEPATH="/apps/modules/modulefiles/compilers:/apps/modules/modulefiles/environment:/apps/modules/modulefiles/libraries:/apps/modules/modulefiles/tools";' export 'MODULEPATH;' '_ModuleTable001_="X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXt9LG1wYXRoQT17Ii9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvY29tcGlsZXJzIiwiL2FwcHMvbW9kdWxlcy9tb2R1bGVmaWxlcy9lbnZpcm9ubWVudCIsIi9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvbGlicmFyaWVzIiwiL2FwcHMvbW9kdWxlcy9tb2R1bGVmaWxlcy90b29scyIsfSxbInN5c3RlbUJhc2VNUEFUSCJdPSIvYXBwcy9tb2R1bGVzL21vZHVsZWZpbGVzL2FwcGxpY2F0aW9uczovYXBwcy9tb2R1bGVzL21vZHVsZWZpbGVzL2NvbXBpbGVyczovYXBwcy9tb2R1bGVzL21vZHVsZWZpbGVz";' export '_ModuleTable001_;' '_ModuleTable002_="L2Vudmlyb25tZW50Oi9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvbGlicmFyaWVzOi9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvdG9vbHMiLH0=";' export '_ModuleTable002_;' '_ModuleTable_Sz_="2";' export '_ModuleTable_Sz_;'
++ MODULEPATH=/apps/modules/modulefiles/compilers:/apps/modules/modulefiles/environment:/apps/modules/modulefiles/libraries:/apps/modules/modulefiles/tools
++ export MODULEPATH
++ _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXt9LG1wYXRoQT17Ii9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvY29tcGlsZXJzIiwiL2FwcHMvbW9kdWxlcy9tb2R1bGVmaWxlcy9lbnZpcm9ubWVudCIsIi9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvbGlicmFyaWVzIiwiL2FwcHMvbW9kdWxlcy9tb2R1bGVmaWxlcy90b29scyIsfSxbInN5c3RlbUJhc2VNUEFUSCJdPSIvYXBwcy9tb2R1bGVzL21vZHVsZWZpbGVzL2FwcGxpY2F0aW9uczovYXBwcy9tb2R1bGVzL21vZHVsZWZpbGVzL2NvbXBpbGVyczovYXBwcy9tb2R1bGVzL21vZHVsZWZpbGVz
++ export _ModuleTable001_
++ _ModuleTable002_=L2Vudmlyb25tZW50Oi9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvbGlicmFyaWVzOi9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvdG9vbHMiLH0=
++ export _ModuleTable002_
++ _ModuleTable_Sz_=2
++ export _ModuleTable_Sz_
++ : -s sh
+ eval
+ module use /gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/modules/all/
++ /apps/modules/LMOD/7.7/lmod/lmod/libexec/lmod bash use /gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/modules/all/
+ eval 'MODULEPATH="/gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/modules/all:/apps/modules/modulefiles/compilers:/apps/modules/modulefiles/environment:/apps/modules/modulefiles/libraries:/apps/modules/modulefiles/tools";' export 'MODULEPATH;' '_ModuleTable001_="X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXt9LG1wYXRoQT17Ii9ncGZzL3Byb2plY3RzL2JzYzMyL3NvZnR3YXJlL3JoZWwvNy40L3BwYzY0bGUvUE9XRVI5L21vZHVsZXMvYWxsIiwiL2FwcHMvbW9kdWxlcy9tb2R1bGVmaWxlcy9jb21waWxlcnMiLCIvYXBwcy9tb2R1bGVzL21vZHVsZWZpbGVzL2Vudmlyb25tZW50IiwiL2FwcHMvbW9kdWxlcy9tb2R1bGVmaWxlcy9saWJyYXJpZXMiLCIvYXBwcy9tb2R1bGVzL21vZHVsZWZpbGVzL3Rvb2xzIix9LFsic3lzdGVtQmFzZU1QQVRIIl09Ii9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvYXBwbGlj";' export '_ModuleTable001_;' '_ModuleTable002_="YXRpb25zOi9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvY29tcGlsZXJzOi9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvZW52aXJvbm1lbnQ6L2FwcHMvbW9kdWxlcy9tb2R1bGVmaWxlcy9saWJyYXJpZXM6L2FwcHMvbW9kdWxlcy9tb2R1bGVmaWxlcy90b29scyIsfQ==";' export '_ModuleTable002_;' '_ModuleTable_Sz_="2";' export '_ModuleTable_Sz_;'
++ MODULEPATH=/gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/modules/all:/apps/modules/modulefiles/compilers:/apps/modules/modulefiles/environment:/apps/modules/modulefiles/libraries:/apps/modules/modulefiles/tools
++ export MODULEPATH
++ _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXt9LG1wYXRoQT17Ii9ncGZzL3Byb2plY3RzL2JzYzMyL3NvZnR3YXJlL3JoZWwvNy40L3BwYzY0bGUvUE9XRVI5L21vZHVsZXMvYWxsIiwiL2FwcHMvbW9kdWxlcy9tb2R1bGVmaWxlcy9jb21waWxlcnMiLCIvYXBwcy9tb2R1bGVzL21vZHVsZWZpbGVzL2Vudmlyb25tZW50IiwiL2FwcHMvbW9kdWxlcy9tb2R1bGVmaWxlcy9saWJyYXJpZXMiLCIvYXBwcy9tb2R1bGVzL21vZHVsZWZpbGVzL3Rvb2xzIix9LFsic3lzdGVtQmFzZU1QQVRIIl09Ii9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvYXBwbGlj
++ export _ModuleTable001_
++ _ModuleTable002_=YXRpb25zOi9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvY29tcGlsZXJzOi9hcHBzL21vZHVsZXMvbW9kdWxlZmlsZXMvZW52aXJvbm1lbnQ6L2FwcHMvbW9kdWxlcy9tb2R1bGVmaWxlcy9saWJyYXJpZXM6L2FwcHMvbW9kdWxlcy9tb2R1bGVmaWxlcy90b29scyIsfQ==
++ export _ModuleTable002_
++ _ModuleTable_Sz_=2
++ export _ModuleTable_Sz_
++ : -s sh
+ eval
+ set +x
The following have been reloaded with a version change:
1) NCO/4.4.9-foss-2018b => NCO/4.7.2-foss-2018b
+ export PYTHONPATH=/esarchive/scratch/ruben/earthdiagnostics:/gpfs/projects/bsc32/software/rhel/7.5/ppc64le/POWER9/software/earthdiagnostics/3.1.0-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.5/ppc64le/POWER9/software/pynco/0.0.3-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.5/ppc64le/POWER9/software/numpy/1.15.0-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.5/ppc64le/POWER9/software/diagonals/0.1.0-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.5/ppc64le/POWER9/software/numba/0.40.1-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/software/Iris/2.2.0-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/software/Cartopy/0.16.0-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/software/GEOS/3.6.2-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/software/ecCodes/2.7.3-foss-2018b-Python-2.7.15/lib/python2.7/site-packages
+ PYTHONPATH=/esarchive/scratch/ruben/earthdiagnostics:/gpfs/projects/bsc32/software/rhel/7.5/ppc64le/POWER9/software/earthdiagnostics/3.1.0-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.5/ppc64le/POWER9/software/pynco/0.0.3-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.5/ppc64le/POWER9/software/numpy/1.15.0-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.5/ppc64le/POWER9/software/diagonals/0.1.0-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.5/ppc64le/POWER9/software/numba/0.40.1-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/software/Iris/2.2.0-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/software/Cartopy/0.16.0-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/software/GEOS/3.6.2-foss-2018b-Python-2.7.15/lib/python2.7/site-packages:/gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/software/ecCodes/2.7.3-foss-2018b-Python-2.7.15/lib/python2.7/site-packages
+ /esarchive/scratch/ruben/earthdiagnostics/earthdiagnostics/earthdiags.py -lc INFO -f /esarchive/scratch/ruben/shock_comparison/processing_data/ec_flow/a1qw_suite/conf/19580201/a1qw_diagnostics_19580201_4_57.conf
ERROR:numba.cuda.cudadrv.driver:Call to cuInit results in CUDA_ERROR_NO_DEVICE
/gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/software/Iris/2.2.0-foss-2018b-Python-2.7.15/lib/python2.7/site-packages/scitools_iris-2.2.0-py2.7.egg/iris/fileformats/cf.py:320: UserWarning: Missing CF-netCDF auxiliary coordinate variable u'time_centered', referenced by netCDF variable u'sic'
warnings.warn(message % (name, nc_var_name))
/gpfs/projects/bsc32/software/rhel/7.4/ppc64le/POWER9/software/Iris/2.2.0-foss-2018b-Python-2.7.15/lib/python2.7/site-packages/scitools_iris-2.2.0-py2.7.egg/iris/fileformats/cf.py:645: UserWarning: Missing CF-netCDF label variable u'time_centered', referenced by netCDF variable u'sic'
warnings.warn(message % (name, nc_var_name))
Exception TypeError: "'NoneType' object is not callable" in <bound method ModuleRef.__del__ of <llvmlite.binding.module.ModuleRef object at 0x7fff85bea990>> ignored
+ wait
+ touch /esarchive/scratch/ruben/shock_comparison/processing_data/ec_flow/a1qw_suite/suite_home/a1qw_diagnostics/19580201/Member_4/Chunks_57/run_custom.job1.completed
+ trap 0
+ exit 0