GRIB data loading
The issue is for GRIB data loading for CERISE project. I summarize what we have now.
There are several options for reading GRIB.
[R approach]
-
gribr: A wrapper of ecCodes developed by ECMWF
We've created a function using this package to load files, with proper reshaping and attributes. But it has a problem loading system5 data. The maintainer has been contacted but we don't know when the problem can be solved. -
rNOMADS: A package developed by NOAA
The first try showed too many errors. I thought it must be a full workflow from downloading GRIBS from NOAA website then reading the files, and I got connection errors. But I need to give it a try again to see if it can load local files. - raster: Have difficulty retrieving metadata(?) Haven't explored much yet
- Rgetgrib: not sure if it is maintained still
[Python approach]
-
pygrib: we have a python function using pygrib and an R function to use the python function (by
reticulate
.) It works well for system5 data. The attribute retrieval and multiple file loading need to be developed. - xarray: Have more advanced functionality compared to pygrib. For example, reading multiple files at once. Need to explore more to see how to transfer the xarray object to R object.
The two things we have now, gribr and pygrib, they are quite similar since they're the basic API of ecCodes. They both can only load one file per time, which is not a big problem since we can load them one after another and put into one big array. But if xarray can take care of the whole thing, it is a better option.
Another thing is, if we can test MARS machine directly, we may be able to get the GRIB files in the structure we want, so our function may not need to cover all the possibilities regarding file structure. If the problem of gribr loading system5 data is due to multiple members and we can get the files per member, the gribr wrapper function we have now may work well already.
The GRIB files we use to test are under:
system5: /esarchive/exp/ecmwf/system5_m1/original_files/fcmean_od_sfc_msmm_ecmf
era5: /esarchive/recon/ecmwf/era5/original_files/an_era5_sfc_moda_ecmf
Please let me know if there is anything missing, thanks!
Cheers,
An-Chi