GFAS shapefiles creation sometime fails.
mguevara @mguevara · 1 week ago @ctena, I am also having problems with HERMESv3_GR. In this case, it is an issue I already found in the past. When creating the shapefile for gfas emissions, some of the shapefiles parts are not correctly created. I tried to run the model with 50, 100 and 125 cpus, but everytime there is a different piece of the shapefile that is not created. For instance, with 50 cpus, I get this error:
Traceback (most recent call last):
File "hermes.py", line 151, in <module>
run()
File "hermes.py", line 144, in run
date = HermesGr(Config()).main()
File "hermes.py", line 106, in main
ei.do_regrid()
File "/gpfs/scratch/cns51/cns51404/a53a/HERMESv3/sources/HERMESv3_GR/hermesv3_gr/modules/emision_inventories/point_gfas_emission_inventory.py", line 265, in do_regrid
grid_shp = self.grid.to_shapefile(full_grid=False).reset_index()
File "/gpfs/scratch/cns51/cns51404/a53a/HERMESv3/sources/HERMESv3_GR/hermesv3_gr/modules/grids/grid.py", line 439, in to_shapefile
gdf.to_file(self.shapefile_path)
File "/apps/PYTHON/3.7.4_ES/INTEL/lib/python3.7/site-packages/geopandas/geodataframe.py", line 429, in to_file
to_file(self, filename, driver, schema, **kwargs)
File "/apps/PYTHON/3.7.4_ES/INTEL/lib/python3.7/site-packages/geopandas/io/file.py", line 130, in to_file
schema=schema, **kwargs) as colxn:
File "/apps/PYTHON/3.7.4_ES/INTEL/lib/python3.7/site-packages/fiona/env.py", line 397, in wrapper
return f(*args, **kwargs)
File "/apps/PYTHON/3.7.4_ES/INTEL/lib/python3.7/site-packages/fiona/__init__.py", line 263, in open
**kwargs)
File "/apps/PYTHON/3.7.4_ES/INTEL/lib/python3.7/site-packages/fiona/collection.py", line 162, in __init__
self.session.start(self, **kwargs)
File "fiona/ogrext.pyx", line 1100, in fiona.ogrext.WritingSession.start
fiona.errors.DriverIOError: Failed to create file /gpfs/scratch/cns51/cns51404/a53a/HERMESv3/auxiliar_files/d01/shapefiles_n50/grid_shapefile_48.shp: No such file or directory
+ exit 1
And if you go to /gpfs/scratch/cns51/cns51404/a53a/HERMESv3/auxiliar_files/d01/shapefiles_n50/
, you see that the piece 48 is missing, together with others. In some other runs only a few pieces are missing, or even only one, but it seems impossible to have them all. How can I solve this problem?
cns51404@login1:/gpfs/scratch/cns51/cns51404/a53a/HERMESv3/auxiliar_files/d01/shapefiles_n50> ls
grid_shapefile_0.cpg grid_shapefile_26.prj grid_shapefile_30.shx grid_shapefile_34.dbf grid_shapefile_36.shp grid_shapefile_42.cpg grid_shapefile_44.prj grid_shapefile_46.shx
grid_shapefile_0.dbf grid_shapefile_26.shp grid_shapefile_31.cpg grid_shapefile_34.prj grid_shapefile_36.shx grid_shapefile_42.dbf grid_shapefile_44.shp grid_shapefile_47.cpg
grid_shapefile_0.prj grid_shapefile_26.shx grid_shapefile_31.dbf grid_shapefile_34.shp grid_shapefile_37.cpg grid_shapefile_42.prj grid_shapefile_44.shx grid_shapefile_47.dbf
grid_shapefile_0.shp grid_shapefile_27.cpg grid_shapefile_31.prj grid_shapefile_34.shx grid_shapefile_37.dbf grid_shapefile_42.shp grid_shapefile_45.cpg grid_shapefile_47.prj
grid_shapefile_0.shx grid_shapefile_27.dbf grid_shapefile_31.shp grid_shapefile_35.cpg grid_shapefile_37.prj grid_shapefile_42.shx grid_shapefile_45.dbf grid_shapefile_47.shp
grid_shapefile_24.cpg grid_shapefile_27.prj grid_shapefile_31.shx grid_shapefile_35.dbf grid_shapefile_37.shp grid_shapefile_43.cpg grid_shapefile_45.prj grid_shapefile_47.shx
grid_shapefile_24.dbf grid_shapefile_27.shp grid_shapefile_33.cpg grid_shapefile_35.prj grid_shapefile_37.shx grid_shapefile_43.dbf grid_shapefile_45.shp
grid_shapefile_24.prj grid_shapefile_27.shx grid_shapefile_33.dbf grid_shapefile_35.shp grid_shapefile_40.cpg grid_shapefile_43.prj grid_shapefile_45.shx
grid_shapefile_24.shp grid_shapefile_30.cpg grid_shapefile_33.prj grid_shapefile_35.shx grid_shapefile_40.dbf grid_shapefile_43.shp grid_shapefile_46.cpg
grid_shapefile_24.shx grid_shapefile_30.dbf grid_shapefile_33.shp grid_shapefile_36.cpg grid_shapefile_40.prj grid_shapefile_43.shx grid_shapefile_46.dbf
grid_shapefile_26.cpg grid_shapefile_30.prj grid_shapefile_33.shx grid_shapefile_36.dbf grid_shapefile_40.shp grid_shapefile_44.cpg grid_shapefile_46.prj
grid_shapefile_26.dbf grid_shapefile_30.shp grid_shapefile_34.cpg grid_shapefile_36.prj grid_shapefile_40.shx grid_shapefile_44.dbf grid_shapefile_46.shp
Carles Tena
Yes, this is the random MPI freeze error.
How did you proceed after getting this error?
Re-submit the job without intervention Removing the shapefiles_nXX directory Removing the content of the shapefiles_nXX folder Normally I recommend to go throw the 3rd option. The 1st one may work but I'm not sure but the worst option to do is the second one because the deleted directory should be created again and is in that point of directory creation when we are getting that MPI freeze error.
Carles
mguevara
@ctena , I tried all three options and the error persist (even when just removing the content of the shapefiles_nXX folder). HERMESv3_GR is not capable of creating all the shapefiles needed. For instance, in my last attempt the model missed to create 3 pieces of the shapefile. Is there any other solution?
Carles Tena
Not in a short term. That is why we have 5 (or more) retrials for the HERMESv3_GR_PREPROC job.
Long term solutions may be:
- Change writing/reading method of the shapefiles
- Try to solve the freezing error substituting Barriers by filelock (see #30 (closed) )
mguevara
OK, can you add this issue as a priority in the AC-HERMES tag? (if it's not already there)