working_groups:ces:postprocess []

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== AC Post-Process Working Group Meetings ======

===== 20230502 =====
Attendants: 

**NES**:
  * Alba requested **Pypi** to remove the current NES package. Stattus?
  * **Unit changing** (Dene's package)
    * Pending to provide examples to Dene. [[https://earth.bsc.es/gitlab/ac/unit-converter/-/issues/1|unit-converter #1]]
  * **Rotated-Nested** create. Pending to merge [[https://earth.bsc.es/gitlab/es/NES/-/issues/46|#46]]
  * **MONARCH** output type
    * Pending: **CMAQ** and **WRF-Chem** [[https://earth.bsc.es/gitlab/es/NES/-/issues/63|#63]]
  * **Tutorial** date selected: [[https://doodle.com/meeting/participate/id/bmY89D3b|Tuesday 2023/05/16 from 15:00 to 17:00]]
    * 23/28 people
    * **TODO**: Check & refactor Jupyther Notebooks [[https://earth.bsc.es/gitlab/es/NES/-/issues/65|#65]]
  * **Pending**:
    * **Unmapped** horizontal interpolation regions [[https://earth.bsc.es/gitlab/es/NES/-/issues/64|#64]]

**SNES**:
  * Added **daily** input type form ESARCHIVE
  * Added **conservative** horizontal interpolation
    * TODO: agree and implement flux to mass (and mass to flux) for conservative remapping.
      * We have suggested some options and the chosen one from Oriol is to add a new column in the CSV indicating if you want to apply that transformation. [[https://earth.bsc.es/gitlab/es/SNES/-/issues/38|#38]]

**auto-SNES**:
  * Oriol is **testing the conservative** remapping with auto-SNES & AS/4.
  * Dani Beltran suggest to remove the current AS/4 to point to the new tag, but I rejected the idea until auto-SNES is adapted to the new tag changes. [[https://earth.bsc.es/gitlab/es/autosubmit/-/issues/1022|Autosubmit #1022]]
  * **Pending**:
    * Testing suite [[https://earth.bsc.es/gitlab/es/auto-SNES/-/issues/8|#8]]

**Other stuff**
  * **Output-checker**: interpolated data stattus? CAMS2_40 check? [[https://earth.bsc.es/gitlab/es/auto-monarch/-/issues/532|auto-MONARCH #532]]
  * Slow multi-day statistics (Miriam)
  * Maria's monthly outputs [[https://earth.bsc.es/gitlab/ac/nmmb-monarch/-/issues/224|MONARCH #224]]
  * Providentia Interpolation [[https://earth.bsc.es/gitlab/es/SNES/-/issues/17|SNES #17]]


===== 20230404 =====
Attendants: 

**NES**
  * Alba suggests opening the NES library with PyPi:
    * Readthedocs documentation
    * Change the name to one not picked in the PyPi repository
      * **PIOLES**: **P**arallel **I**nput **O**utput **L**ibrary for **E**arth **S**cience
      * PIOLib
      * PIOLIbES
      * **APIO**: **A**utomatic **P**arallel **I**nput **O**utput
      * **PIOESlib**: **P**arallel **IO** for **E**arth **S**cience **Lib**rary
      * **MIOPES**: **M**ulti **IO** **P**arallel for **E**arth **S**cience
  * Unit changing (Dene's package)
    * Alba is going to lead that task.
  * 


**SNES**
  * No news

**auto-SNES**
  * No news

**Other stuff**
  * Slow multi-day statistics (Miriam)
    * Maria's monthly outputs
    * Providentia Interpolation


===== 20230321 =====
Attendants: 

**NES**
  * DONE:
    * Bugfix on cell_methods serial write 
    * Write 2D string data (timezones)
    * Sum Nes objects (sum variables)
    * Write by time step
    * Concatenation improved times
  * Working on:
    * Spin-up bug
  * TODO:
    * Nested-Rotated domain (needed for HERMES)
    * Spatial Join: Improve time & memory for large datasets (needed for HERMES)
  * Training requested.

**SNES**
  * No news

**Auto-SNES**
  * Testing suite experiments created.
  * Next steps:
    * Fix the benchmark experiments (waiting for auto-MONARCH validation)

**Auto-MONARCH**
  * Testing Suite. Output checker fails for NES_master

**Other stuff**
  * 

===== 20230307 =====
Attendants: 

**NES**
  * v1.1.0 release done:
    * Improve Lat-Lon to Cartesian coordinates method (used in Providentia).
    * Horizontal Interpolation: Conservative
    * Shapefile:
      * Function **to_shapefile()** to create shapefiles from a NES object without losing data from the original grid and being able to select the time and level.
      * Function **from_shapefile()** to create a new grid with data from a shapefile after doing a spatial join.
      * Function **create_shapefile()** can now be used in parallel.
    * Function **calculate_grid_area()** to calculate the area of each cell in a grid.
    * Function **calculate_geometry_area()** to calculate the area of each cell given a set of geometries.
    * Function **get_spatial_bounds_mesh_format()** to get the lon-lat boundaries in a mesh format (used in pcolormesh).
    * Bugs fixing:
      * Correct the dimensions of the resulting points datasets from any interpolation.
      * Amend the interpolation method to take into account the cases in which the distance among points equals zero.
      * Correct the way we retrieve the level positive value.
      * Correct how to calculate the spatial bounds of LCC and Mercator grids: the dimensions were flipped.
      * Correct how to calculate the spatial bounds for all grids: use read axis limits instead of write axis limits.
      * Calculate centroids from coordinates in creating shapefiles, instead of using the geopandas function 'centroid', which raises a warning for possible errors.
      * Enable selection of variables on the creation of shapefiles.
      * Correct read and write parallel limits.
      * Correct data type in the parallelization of points datasets.
      * Correct errors that appear when trying to select coordinates and write the file.
  * Working on:
    * Pending to merge:
      * Bug on cell_methods serial write 
  * Next steps:
    * Sum Nes objects (sum variables)
    * Write 2D string data (timezones)
    * Nested-Rotated domain
  * Training requested.

**SNES**
  * v1.1.0 release done:
    * Correct way to use level.positive NetCDF metadata.
    * Added sum_levels new options in the CSV.
      * Added the option to choose if you want to enable (or not) the sum_levels option by variable with the 'sum_levels' column.
      * Added the option to choose if you want to write (or not) the sum_levels option by variable with the 'sum_levels_path' column
    * Added NES conservative horizontal interpolation.
    * NES update to v1.1.0 (see NESv1.1.0 release notes)
  * Next steps:
    * run an auto-MONARCH testing suite

**Auto-SNES**
  * Next steps:
    * Testing suite:
      * Create a testing suite with operational runs (at least)

**Auto-MONARCH**
  * Everything OK?

**Other stuff**
  * 


===== 20230207 =====
Attendants: 

**NES**
  * Done:
    * Bugfixing:
      * Correct the dimensions of the resulting points datasets from any interpolation.
      * Amend the interpolation method to account for cases in which the distance among points equals zero.
      * Correct the way we retrieve the level positive value.
      * Correct how to calculate the spatial bounds of LCC and Mercator grids: the dimensions were flipped.
      * **Pending to merge-request**:
        * Correct how to calculate the spatial bounds for all grids: use read axis limits instead of write axis limits.
        * Calculate centroids from coordinates in the creation of shapefiles, instead of using the geopandas function 'centroid', which raises a warning for possible errors.
      * Enable selection of variables on the creation of shapefiles
    * New Features:
      * Improve Lat-Lon to Cartesian coordinates method (used in Providentia).
      * Function to_shapefile() to create shapefiles from a NES object without losing data from the original grid and being able to select the time and level.
      * Function from_shapefile() to create a new grid with data from a shapefile after doing a spatial join.
      * Function create_shapefile() can now be used in parallel.
      * **Pending to merge**:
        * Function calculate_grid_area() to calculate the area of each cell in a grid.
        * Function calculate_geometry_area() to calculate the area of each cell given a set of geometries.
  * Working on:
    * Horizontal Interpolation: Conservative
  * Todo:
    * Release v1.1.0:
      * Finish Conservative remapping
      * Add Tests.
    * HERMES's needs:
      * Write by adding timesteps

**SNES**
  * Waiting for **NES** release.

**Auto-SNES**
  * Waiting for **SNES** release.

**Auto-MONARCH**
  * HERMES on CIRRUS:
    * geopandas version <0.11

**Other stuff**
  * CALIOPE_CAT:
    * Barnes cross-validation.
      * Code adapted to achieve the cross-validation but with Errors that I've not been able to solve.
        * It cannot run in a loop of stations. 
        * Single station works fine. Then concatenate the results.
      * Miriam is testing the one-month execution (I've only tested one day)

===== 20230124 =====
Attendants: 

**NES**
  * Working on conservative horizontal interpolation
  * New: to_shapefile & from_fhapefile
  * TODO:
    * Tests for merging

**SNES**
  * Vertical direction set
  * Only interpolation

**Auto-Monarch**
  * 

**Auto-SNES**
  * Waiting for SNES release


**Other stuff**
  * 

===== 20221213 =====
Attendants: 

**NES**
  * Solved some bugs but this does not affect any operational operation.
  * Vertical direction hardcoded in NES

**SNES**
  * How do we threat the vertical dimension?

**Auto-Monarch**
  * CAMS RA workflow?

**Auto-SNES**
  * Working on documentation and test cases

**BDRC**
  * It should be working before February 2023
  * New needs. 12h more.
    * SNES in serial.
    * To speedup an intermediate parallelization strategy has to be applied. This is the [[http://example.com|SNES issue]] to follow up on this improvement.
  * CIRRUS installation.
    * This week

**Other stuff**
  * Providentia interpolation:
    * Bug solved.
    * Release 1.0.0
    * [[https://nextcloud.bsc.es/index.php/apps/onlyoffice/27916485?filePath=%2FAC-PostProcess%2FProvidentiaInterpolation_2_SNES.xlsx|Roadmap of auto-SNES transition]] done.

===== 20221129 =====
Attendants: Francesca, Francesco, Carles, Alba, Alejandro, Gilbert, Miriam

**NES**
  * CAMS RA format has been validated by INERIS.
  * v1.0.0 version has been released.
  * NESv1.0.0 is installed as a nord3v2 module

**SNES**
  * **production** branch updated with CAMS_RA format
  * Created v1.0.0 version release.


**Auto-Monarch**
  * Alejandro will start the CAMS RA workflow
    * SNES branch should be updated to tag v.1.0.0
    * SNES modules should be updated in order to use the installed module

**Auto-SNES**
  * Working on documentation and test cases

**Other stuff**
  * Providentia interpolation:
    * Alba is going to start the climatology on Providentia Interpolation project.
    * In parallel she will analyze the functionalities in order to get all the Providentia Interpolation current features
    * That current features will be used as a requirement for SNES future release.
    * Miriam will help to get the requirements list with their experiences using it.
    * She will also add suggestions for improvements.
    * If the priority list does not change, SNES with Providentia Interpolation features will be postponed till MN5 arrive. 


===== 20221018 =====
Attendants: Francesca, Francesco, Carles, Alba, Alejandro, Gilbert, Miriam

**NES**
  * Horizontal interpolation differences comes from normalization of weights. 
  * Tests on coordinates bounds in parallel.


**Auto-Monarch**
  * implemented in the workflow
  * TODO: remove split_method from the template


===== 20221004 =====
Attendants: Francesca, Francesco, Carles, Alba, Alejandro, Gilbert, Miriam


**NES**
  * Changing labels related to AC-CES to CES to avoid discussion on AC-CES group (Alba)

**SNES**
  * BDRC [[https://earth.bsc.es/gitlab/es/SNES/-/issues/18|#18]]
    * Grib2 does not accept NaN. If FillFirstHour fails I'm substituting NaN by zero
    * Current Grib2 contains negative values
<file>
grib_dump /esarchive/scratch/fbeninca/SNES/2022091212_3H_SDSWAS_NMMB-BSC-v2_OPER.grb2 >> /gpfs/scratch/bsc32/bsc32538/SNES_tests/OUT/monarch/BDRC_AEMET/grib2/orig_dump.txt

cat /gpfs/scratch/bsc32/bsc32538/SNES_tests/OUT/monarch/BDRC_AEMET/grib2/orig_dump.txt | grep minimum
[bsc32538@login4 grib2]$ cat orig_dump.txt | grep minimum
  #-READ ONLY- minimum = -3.87477e-33;
  #-READ ONLY- minimum = -6.30957e-19;
  #-READ ONLY- minimum = -1.63499e-30;
  #-READ ONLY- minimum = -3.06059e-29;
  ...

</file>
    * SNES implementation done. Pending tasks:
      * Testing
      * Auto-MONARCH:
        * File gathering 
          * NetCDF
          * Grib2
        *md5sum

  * Move project from ES group to AC group in Gitlab (Alba)

**Auto-Monarch**
  * Memory issue running SNES in a real case of CAMS2_40 forecast. Split in 2 jobs?

**Auto-SNES**

**Other stuff**


===== 20220920 =====
Attendants: Francesca, Francesco, Carles, Alba, Alejandro, Gilbert, Miriam


**NES**
  * Installation MN4
  * Selection function (FillFirstHour & GFAS-Hourly)

**SNES**
  * fill_first_hour_from [[https://earth.bsc.es/gitlab/es/SNES/-/issues/20|#20]]
  * Select write by variable [[https://earth.bsc.es/gitlab/es/SNES/-/issues/21|#21]]
  * BDRC [[https://earth.bsc.es/gitlab/es/SNES/-/issues/18|#18]]
    * YES: File per variable (netcdf & grib2) -> template gather them ?
    * NO: Or... Grib2 - One file with all variables [[https://earth.bsc.es/gitlab/es/SNES/-/issues/19|#19]]

**Auto-Monarch**
* DUST OK
* WIP CAMS2_40

**Auto-SNES**

**Other stuff**
  * CES meeting presentation
===== 20220906 =====

Attendants: Francesca, Francesco, Carles, Alba, Alejandro, Gilbert

**NES**

  * Providentia experiment format (Alba)
  * Create shapefile (Alba)
  * Adding shapefile info to NetCDF (Alba)
  * Selecting function WIP (GFAS-Hourly Carles)

**SNES**

  * Enza ask for member implementation.

**Auto-Monarch**

  * How it is going on?

**Auto-SNES**

  * WIP: Pending issue to be resolved [[https://earth.bsc.es/gitlab/es/autosubmit/-/issues/861|autosubmit #861]]

**Other stuff**

  * BDRC: Outside BSC post-process
  * fill_first_hour
User Tools

Site Tools

Page Tools
