CDORemap: multiple cores and weight file as input
To speed up the CDO process, cdo option -P can be used to request a specific number of OpenMP threads. The development has been done by @cdelgado in branch develop-CDORemap-ncores (thanks!) Here are some notes about how it works.
- The setting that shows a better performance with
-P 8is: On Nord3v2 compute node (interactive session). 1 core: 33s / 8 cores: 16s - On WS, 8 cores and 1 core make no difference.
- Only some operators support
-P. See https://code.mpimet.mpg.de/projects/cdo/wiki/OpenMP_support. And it also depends on how CDO module is compiled.cdo -Vto see if the module supports openMP (should be listed under "Features")
Even -P doesn't work for all the cases, we can still include it as long as it doesn't slow down or break the process.
Another thing we can test is using a weight file to do the regridding. The reference is here: https://code.mpimet.mpg.de/boards/1/topics/10290. Generate weight.nc by cdo -genbil,target_grid infile1.nc weight.nc for the first time, then use cdo -remap,target_grid,weights.nc infile.nc outfile.nc for all the files.
It could be useful when several files need to be interpolated (e.g., startR) However, the person in the discussion said that the method didn't improve the time efficiency. We can do some tests to see if the weight method improves time or memory efficiency. If not, we forget about this development.
FYI @vagudets