Attendants:

NES:

• Alba requested Pypi to remove the current NES package. Stattus?
• Unit changing (Dene's package)
  • Pending to provide examples to Dene. unit-converter #1
• Rotated-Nested create. Pending to merge #46
• MONARCH output type
  • Pending: CMAQ and WRF-Chem #63
• Tutorial date selected: Tuesday 2023/05/16 from 15:00 to 17:00
  • 23/28 people
  • TODO: Check & refactor Jupyther Notebooks #65
• Pending:
  • Unmapped horizontal interpolation regions #64

SNES:

• Added daily input type form ESARCHIVE
• Added conservative horizontal interpolation
  • TODO: agree and implement flux to mass (and mass to flux) for conservative remapping.
    • We have suggested some options and the chosen one from Oriol is to add a new column in the CSV indicating if you want to apply that transformation. #38

auto-SNES:

• Oriol is testing the conservative remapping with auto-SNES & AS/4.
• Dani Beltran suggest to remove the current AS/4 to point to the new tag, but I rejected the idea until auto-SNES is adapted to the new tag changes. Autosubmit #1022
• Pending:
  • Testing suite #8

Other stuff

• Output-checker: interpolated data stattus? CAMS2_40 check? auto-MONARCH #532
• Slow multi-day statistics (Miriam)
• Maria's monthly outputs MONARCH #224
• Providentia Interpolation SNES #17

Attendants:

NES

• Alba suggests opening the NES library with PyPi:
  • Readthedocs documentation
  • Change the name to one not picked in the PyPi repository
    • PIOLES: Parallel Input Output Library for Earth Science
    • PIOLib
    • PIOLIbES
    • APIO: Automatic Parallel Input Output
    • PIOESlib: Parallel IO for Earth Science Library
    • MIOPES: Multi IO Parallel for Earth Science
• Unit changing (Dene's package)
  • Alba is going to lead that task.

SNES

• No news

auto-SNES

• No news

Other stuff

• Slow multi-day statistics (Miriam)
  • Maria's monthly outputs
  • Providentia Interpolation

Attendants:

NES

• DONE:
  • Bugfix on cell_methods serial write
  • Write 2D string data (timezones)
  • Sum Nes objects (sum variables)
  • Write by time step
  • Concatenation improved times
• Working on:
  • Spin-up bug
• TODO:
  • Nested-Rotated domain (needed for HERMES)
  • Spatial Join: Improve time & memory for large datasets (needed for HERMES)
• Training requested.

SNES

• No news

Auto-SNES

• Testing suite experiments created.
• Next steps:
  • Fix the benchmark experiments (waiting for auto-MONARCH validation)

Auto-MONARCH

• Testing Suite. Output checker fails for NES_master

Other stuff

Attendants:

NES

• v1.1.0 release done:
  • Improve Lat-Lon to Cartesian coordinates method (used in Providentia).
  • Horizontal Interpolation: Conservative
  • Shapefile:
    • Function to_shapefile() to create shapefiles from a NES object without losing data from the original grid and being able to select the time and level.
    • Function from_shapefile() to create a new grid with data from a shapefile after doing a spatial join.
    • Function create_shapefile() can now be used in parallel.
  • Function calculate_grid_area() to calculate the area of each cell in a grid.
  • Function calculate_geometry_area() to calculate the area of each cell given a set of geometries.
  • Function get_spatial_bounds_mesh_format() to get the lon-lat boundaries in a mesh format (used in pcolormesh).
  • Bugs fixing:
    • Correct the dimensions of the resulting points datasets from any interpolation.
    • Amend the interpolation method to take into account the cases in which the distance among points equals zero.
    • Correct the way we retrieve the level positive value.
    • Correct how to calculate the spatial bounds of LCC and Mercator grids: the dimensions were flipped.
    • Correct how to calculate the spatial bounds for all grids: use read axis limits instead of write axis limits.
    • Calculate centroids from coordinates in creating shapefiles, instead of using the geopandas function 'centroid', which raises a warning for possible errors.
    • Enable selection of variables on the creation of shapefiles.
    • Correct read and write parallel limits.
    • Correct data type in the parallelization of points datasets.
    • Correct errors that appear when trying to select coordinates and write the file.
• Working on:
  • Pending to merge:
    • Bug on cell_methods serial write
• Next steps:
  • Sum Nes objects (sum variables)
  • Write 2D string data (timezones)
  • Nested-Rotated domain
• Training requested.

SNES

• v1.1.0 release done:
  • Correct way to use level.positive NetCDF metadata.
  • Added sum_levels new options in the CSV.
    • Added the option to choose if you want to enable (or not) the sum_levels option by variable with the 'sum_levels' column.
    • Added the option to choose if you want to write (or not) the sum_levels option by variable with the 'sum_levels_path' column
  • Added NES conservative horizontal interpolation.
  • NES update to v1.1.0 (see NESv1.1.0 release notes)
• Next steps:
  • run an auto-MONARCH testing suite

Auto-SNES

• Next steps:
  • Testing suite:
    • Create a testing suite with operational runs (at least)

Auto-MONARCH

• Everything OK?

Other stuff

Attendants:

NES

• Done:
  • Bugfixing:
    • Correct the dimensions of the resulting points datasets from any interpolation.
    • Amend the interpolation method to account for cases in which the distance among points equals zero.
    • Correct the way we retrieve the level positive value.
    • Correct how to calculate the spatial bounds of LCC and Mercator grids: the dimensions were flipped.
    • Pending to merge-request:
      • Correct how to calculate the spatial bounds for all grids: use read axis limits instead of write axis limits.
      • Calculate centroids from coordinates in the creation of shapefiles, instead of using the geopandas function 'centroid', which raises a warning for possible errors.
    • Enable selection of variables on the creation of shapefiles
  • New Features:
    • Improve Lat-Lon to Cartesian coordinates method (used in Providentia).
    • Function to_shapefile() to create shapefiles from a NES object without losing data from the original grid and being able to select the time and level.
    • Function from_shapefile() to create a new grid with data from a shapefile after doing a spatial join.
    • Function create_shapefile() can now be used in parallel.
    • Pending to merge:
      • Function calculate_grid_area() to calculate the area of each cell in a grid.
      • Function calculate_geometry_area() to calculate the area of each cell given a set of geometries.
• Working on:
  • Horizontal Interpolation: Conservative
• Todo:
  • Release v1.1.0:
    • Finish Conservative remapping
    • Add Tests.
  • HERMES's needs:
    • Write by adding timesteps

SNES

• Waiting for NES release.

Auto-SNES

• Waiting for SNES release.

Auto-MONARCH

• HERMES on CIRRUS:
  • geopandas version <0.11

Other stuff

• CALIOPE_CAT:
  • Barnes cross-validation.
    • Code adapted to achieve the cross-validation but with Errors that I've not been able to solve.
      • It cannot run in a loop of stations.
      • Single station works fine. Then concatenate the results.
    • Miriam is testing the one-month execution (I've only tested one day)

Attendants:

NES

• Working on conservative horizontal interpolation
• New: to_shapefile & from_fhapefile
• TODO:
  • Tests for merging

SNES

• Vertical direction set
• Only interpolation

Auto-Monarch

Auto-SNES

• Waiting for SNES release

Other stuff

Attendants:

NES

• Solved some bugs but this does not affect any operational operation.
• Vertical direction hardcoded in NES

SNES

• How do we threat the vertical dimension?

Auto-Monarch

• CAMS RA workflow?

Auto-SNES

• Working on documentation and test cases

BDRC

• It should be working before February 2023
• New needs. 12h more.
  • SNES in serial.
  • To speedup an intermediate parallelization strategy has to be applied. This is the SNES issue to follow up on this improvement.
• CIRRUS installation.
  • This week

Other stuff

• Providentia interpolation:
  • Bug solved.
  • Release 1.0.0
  • Roadmap of auto-SNES transition done.

Attendants: Francesca, Francesco, Carles, Alba, Alejandro, Gilbert, Miriam

NES

• CAMS RA format has been validated by INERIS.
• v1.0.0 version has been released.
• NESv1.0.0 is installed as a nord3v2 module

SNES

• production branch updated with CAMS_RA format
• Created v1.0.0 version release.

Auto-Monarch

• Alejandro will start the CAMS RA workflow
  • SNES branch should be updated to tag v.1.0.0
  • SNES modules should be updated in order to use the installed module

Auto-SNES

• Working on documentation and test cases

Other stuff

• Providentia interpolation:
  • Alba is going to start the climatology on Providentia Interpolation project.
  • In parallel she will analyze the functionalities in order to get all the Providentia Interpolation current features
  • That current features will be used as a requirement for SNES future release.
  • Miriam will help to get the requirements list with their experiences using it.
  • She will also add suggestions for improvements.
  • If the priority list does not change, SNES with Providentia Interpolation features will be postponed till MN5 arrive.

Attendants: Francesca, Francesco, Carles, Alba, Alejandro, Gilbert, Miriam

NES

• Horizontal interpolation differences comes from normalization of weights.
• Tests on coordinates bounds in parallel.

Auto-Monarch

• implemented in the workflow
• TODO: remove split_method from the template

Attendants: Francesca, Francesco, Carles, Alba, Alejandro, Gilbert, Miriam

NES

• Changing labels related to AC-CES to CES to avoid discussion on AC-CES group (Alba)

SNES

• BDRC #18
  • Grib2 does not accept NaN. If FillFirstHour fails I'm substituting NaN by zero
  • Current Grib2 contains negative values

grib_dump /esarchive/scratch/fbeninca/SNES/2022091212_3H_SDSWAS_NMMB-BSC-v2_OPER.grb2 >> /gpfs/scratch/bsc32/bsc32538/SNES_tests/OUT/monarch/BDRC_AEMET/grib2/orig_dump.txt

cat /gpfs/scratch/bsc32/bsc32538/SNES_tests/OUT/monarch/BDRC_AEMET/grib2/orig_dump.txt | grep minimum
[bsc32538@login4 grib2]$ cat orig_dump.txt | grep minimum
  #-READ ONLY- minimum = -3.87477e-33;
  #-READ ONLY- minimum = -6.30957e-19;
  #-READ ONLY- minimum = -1.63499e-30;
  #-READ ONLY- minimum = -3.06059e-29;
  ...

• SNES implementation done. Pending tasks:
  • Testing
  • Auto-MONARCH:
    • File gathering
      • NetCDF
      • Grib2
    • md5sum

• Move project from ES group to AC group in Gitlab (Alba)

Auto-Monarch

• Memory issue running SNES in a real case of CAMS2_40 forecast. Split in 2 jobs?

Auto-SNES

Other stuff

Attendants: Francesca, Francesco, Carles, Alba, Alejandro, Gilbert, Miriam

NES

• Installation MN4
• Selection function (FillFirstHour & GFAS-Hourly)

SNES

• fill_first_hour_from #20
• Select write by variable #21
• BDRC #18
  • YES: File per variable (netcdf & grib2) → template gather them ?
  • NO: Or… Grib2 - One file with all variables #19

Auto-Monarch * DUST OK * WIP CAMS2_40

Auto-SNES

Other stuff

• CES meeting presentation

Attendants: Francesca, Francesco, Carles, Alba, Alejandro, Gilbert

NES

• Providentia experiment format (Alba)
• Create shapefile (Alba)
• Adding shapefile info to NetCDF (Alba)
• Selecting function WIP (GFAS-Hourly Carles)

SNES

• Enza ask for member implementation.

Auto-Monarch

• How it is going on?

Auto-SNES

• WIP: Pending issue to be resolved autosubmit #861

Other stuff

• BDRC: Outside BSC post-process
• fill_first_hour