# OBSALL Apps
This repository contains the scripts related to the Observation Apps - components for ground-based/surface (SYNOP), radiosounding (TEMP), and satellite (AMSU-A) obs data) of the Climate Adaptation Digital Twin (Climate DT). The development is done in the frameworks of the Destination Earth initiative.

# Description
The OBSALL application is currently being developed as a set of scripts (written in `bash`, `fortran`, `python`):
1. for pre-processing (with operators of `cdo`, Climate Data Operators, software) gsv_extracted modeled data (over latitude-longitude domain) for selected meteorological variables into hourly time-slices (00...23 UTCs);
2. for extracting (using `sql` with `odp_api` software) selected meteorological variables at hourly time-slices for observations from: (i) ground-based synoptical stations at fixed at the surface geographical locations/points (i.e., 2D: latitude, longitude); (ii) vertical radiosounding of the atmosphere at changing locations/points (3D: latitude, longitude, single pressure level), and (iii) satellite at changing locations/ points (3D: latitude, longitude, multiple pressure levels);
3. for extracting and interpolating (using `cdo` operators - for synop; & expecting/using `polytope` - for radiosounding and satellite) modeled data for the same time-slices for same selected meteorological variables into corresponding locations/points of (synop, radiosounding, satellite) observations, and adding (using `import` with `odb_api`) such data to ODB;
3. for calculating/ producing relevant statistics such as quantile rank histogram statistics and plots (t-test and others to be added)

# Part SYNOP (ground-based/surface observations)
1. `main_synop.sh` - main bash-script to run Apps for observational data for synoptical stations.
2. `gsv_mod_data.sh` - bash-script to run pre-processing (using `cdo` operators) of modelled data extracted with gsv interface over global domain for selected meteorological variables (measured at synoptical station) into hourly time-slices and saving (temporary) to separate hourly nc-files.
3. `synop_obs.sh` - bash-script to run reading and extraction (using `sql` with `odb_api` software) of selected meteorological variables at hourly time-slices for observations from ground-based synoptical stations at fixed at the surface geographical (latitude, longitude) locations over selected geographical domain (or over the globe) and saving (temporary) to dat-file. Note, that list of available  meteorological variables in open multi-year data ODB includes: 91 - total amount of clouds; 108 - sea level pressure; 80 - 1-hour precipitation; 58 - relative humidity; 999 - 10-minute precipitation intensity; 71 - snow depth; 39 - 2m temperature; 40 - dew point temperature; 62 - visibility; 111 - wind direction; 261 - maximum wind gust in last 10 minutes; 221 -surface wind speed.
4. `synop_mod.sh` - bash-script to run reading, extraction and interpolation (using `cdo` operators) of modelled data for selected meteorological variables at the same hourly time-slices to geographical (latitude, longitude) locations of synoptical stations, and saving (temporary) to dat-file, and adding (using `import` with `odb_api`) such data to ODB. Interpolation is preceeded by constructing unstructured grid based on locations of synop stations and calculating weights for such grid. 
5. `graph_mod_obs.py` - python-script to run an internal self-control in calculating differences between observed and modelled data for locations of synoptical stations 
6. `synop_stats.sh` - bash-script to run (addtional `bash`, `fortran`, `python` scripts) calculating/producing:
- (i) rank histograms for all synop stations for 00, 06, 12 and 18 UTCs (with `produce_rank_histograms_all_stations.sh`, using `rank_histograms_one_station.f95` and `sql/import` with `odb_api`), 
- (ii) standard plots for each synop station (with `produce_standard_plots_all_stations.sh` using `plot_quantiles_rankhist.py` and  `sql` with `oadb_api`; see an example of such plot in **Figure 1a**), and 
- (iii) summary rank histograms for all stations (`summary_rank_histograms_all_stations.sh` using `rank_histogram_summary_statistics.f95`, `plot_p_values_map.py`, `plot_rank_hist_sum_all_stations.py` and `sql` with `odb_api`; see examples of such plots in **Figure 1b** and **Figure 1c**). 
- Note, needed input includes: list of synop stations with geographical coordinates, modelled data at stations coordinates, pre-computed quantiles as a function of time of year and rank histogram bootstrap mean square deviation (MSD) values. 
 - ![Semantic description of image](FIGSEXAMPLES/destine_obs_synop_figs.jpg "Plots for 2 m air temperature ")* Plots for 2 metre air temperature (T2m) for period of 1 Jan - 31 Dec 2020: **Figure 1a** - Quantiles/ time series data (00, 06, 12, and 18 UTCs) and rank histogram (combination of 00, 06, 12, and 18 UTCs) for synop station 126736; **Figure 1b** - P-values for T2m quantiles; **Figure 1c** - Rank histogram summary statistics: normalized p-value frequencies and normalized quantile frequencies*

# Part TEMP (radiosounding observations)
1. `main_radsound.sh` - main bash-script to run Apps for observational data for radiosounding.
2. `gsv_radsound_mod_data.sh` - bash-script to run pre-processing (using `cdo` operators) of modelled data extracted with gsv interface over global domain for air temperature at 2 metre (T2) and at 850 hPa pressure level (T850) into time-slices at 00 and 12 UTCs, calculating difference (T2-T850) between these two temperatures, and saving (temporary) to separate nc-files at 00 and 12 UTC time-slices.
3. `radsound_obs.sh` - bash-script to run reading and extraction (using `sql` with `odb_api` software) of above-mentioned air temperatures at 12 hourly time-slices for radiosounding observations at surface and at altitude at geographical (latitude, longitude, pressure level) locations of radiosounding stations over selected geographical domain (or over the globe), calculatig difference between two temperatures, and saving (temporary) to dat-file.
4. `radsound_mod.sh` - bash-script to run reading, extraction and interpolation (using polytope) of modelled data for air temperature at surface and at altitude at the same time-slices to geographical (latitude, longitude, pressure level) locations of radiosounding stations, calculatig difference between two temperatures, and saving (temporary) to dat-file, and adding (using `import` with `odb_api`) such data to ODB.
5. `radsound_stats.sh` - bash-script to run (addtional `bash`, `fortran`, `python` scripts) calculating/producing:
- (i) rank histograms for all radiosounding stations for 00 and 12 UTCs (with `produce_rank_histograms_all_stations.sh`, using `rank_histograms_one_station.f95` and `sql/import` with `odb_api`),
- (ii) standard plots for each radiosounding station (with `produce_standard_plots_all_stations.sh` using `plot_quantiles_rankhist.py` and  `sql` with `oadb_api`; see an example of such plot in **Figure 2a**), and
- (iii) summary rank histograms for all radiosunding stations (`summary_rank_histograms_all_stations.sh` using `rank_histogram_summary_statistics.f95`, `plot_p_values_map.py`, `plot_rank_hist_sum_all_stations.py` and `sql` with `odb_api`; see examples of such plots in **Figure 2b** and **Figure 2c**).
- Note, needed input includes: list of radiosounding stations with geographical coordinates, modelled data at stations coordinates at 2 metre height and at 850 hPha pressure level, pre-computed quantiles as a function of time of year and rank histogram bootstrap mean square deviation (MSD) values.
- ![Semantic description of image](FIGSEXAMPLES/destine_obs_radsound_figs.jpg "Plots for difference between air temperature at 2 m and 850 hPa pressure level: T2-T850")* Plots for difference in air temperatures at 2 metre and at 850 Ha pressure level (T2-T850) for period of 1 Jan - 31 Dec 2010: **Figure 2a** - Quantiles/ time series data and rank histogram (00 and 12 UTCs) for radiosounding station 51431; **Figure 2b** - P-values for T2-T850 quantiles; **Figure 2c** - Rank histogram summary statistics for 00 and 12 UTCs: normalized p-value frequencies and normalized quantile frequencies*

# Part AMSU-A (satellite observations) 
1. `main_amsua.sh` - main bash-script to run Apps for observational data for satellite-based AMSU-A measurements & `set_env.sh`- bash-script which defines variables and functions, and required modules to be loaded and used in the workflow.
2. `gsv_mod_data.sh` - bash-script to run pre-processing (using `cdo` operators; in later versions - `polytope`) of 2D on surface (`sfc`) and 3D modelled data on pressure levels (`pl`) extracted with gsv interface over global domain. The `Radiance Simulator` (using `convert_pl2ml.py`) interpolates modelled data from `pl`-levels to “model”-levels (note, workflow will initially be demonstrated on a single selected location i.e., around a point of 70.0N, 45.0E in the Barents Sea; and later geographically expanded using `polytope`) and preprocessed data saved in (temporary) hourly GRIB-files. Required 3D variables are: `t` - temperature (130), `q` - specific humidity (133), `clwc` - specific liquid water content (246); and 2D variables are: `skt` - skin temperature (235), `ci` - sea ice fraction (31), `lsm` - land-sea mask (172),`sp` - surface pressure (134), `10u` and `10v` - U and V components of wind speed at 10 m (165 and 166), `2t` - air temperature at 2 m (167), `2d` - dewpoint temperature at 2 m (168), and `z` - geopotential (for orography) (129).
3. `amsua_obs.sh` - bash-script to prepare a metadata file for computation of AMSU-A model counterparts from preprocessed hourly GSV data. It runs depending on the type of climate simulation: (i) reality following data assimilation-like, and (ii) free running climate simulation. For (i) the one-to-one model counterparts can be produced, so actual metadata from AMSU-A observations in the vicinity of the selected location are extracted from ODB. This is done with `filter_data_v3.py` For (ii) the model output can be compared to observations only in climatological terms, so metadata is written only for the centre point of the selected area.
4. `amsua_mod.sh` - bash-script to run the `Radiance Simulator` tool with pre-described control configuration parameters and to produce a model counterpart (file in netcdf-format), which (using `convert_netcdf2txt.py`) is converted to (temporary) dat-file and added (using `import` with `odb_api`) to ODB.
5. The monitoring for AMSU-A is under development. Initially it was done on Puhti HPC of CSC, and then, to be migrated, tested, and implemented into workflow on Lumi HPC of CSC. The required polytope tool is expected to be released in December 2023. For AMSU-A monitoring, `amsua_stats.sh` bash-script is used to run (additional `bash`, `fortran`, `python` scripts) calculating/producing similar statistics as for TEMP, using daily mean values inside the selected area.

# Installation
**Git**: to copy the "obsall" repository to your local directory via Git using:
```
cd your_local_directory
git clone https://earth.bsc.es/gitlab/digital-twins/de_340/obsall.git
cd obsall
git checkout newbranch
```
This will clone the "newbranch"-branch of the "obsall" repository.

**Modules**: to set environment in order to run OBSALL application in LUMI:
```
# Load modules for OBSALL Apps
module load LUMI/23.03
module load partition/C
module load PrgEnv-gnu
module load ecCodes/2.32.0-cpeCray-23.03.lua
module load odb_api/0.18.1-cpeCray-23.03.lua
module load python-climatedt/3.11.3-cpeCray-23.03.lua
```
**Data request:** file `request.yml` is needed for extracting selected modeled data 
(example is for SYNOP Part, where only data at `sfc` is needed) 
<Details>

```
EXPERIMENT:
   DATELIST: 20200201 #Startdate
   MEMBERS: fc0
   CHUNKSIZEUNIT: day   #unit length of the chunk (hour, day, month, year)
   CHUNKSIZE: 1         #number of unit lengths per chunk
   NUMCHUNKS: 1         #chunkc to be run
   CALENDAR: standard   #including/ excluding leap years

GSVREQUEST: #raw data
   domain: g
   class: rd
   type: fc
   expver: hz9n
   stream: lwda
   anoffset: 9
   levtype: "sfc"
   date: "%CHUNK_START_DATE%/to/%CHUNK_SECOND_TO_LAST_DATE%"
   time: 0000/to/2300/by/0100  # every hour
   step: ["0"] # Irrelevant. step will be recalculated in workflow
   param: #["2t"]
   grid: 1/1
   method: nn

APP:
   other: "empty"

OPAREQUEST:
  NSTATS: "1"    # number of variables to request
  1:
    variable: "2t"
    stat: "hourly"
    stat_freq: "hourly"
    output_freq: "daily"
    time_step: 60 # in minutes, 60*timestep length in hours TODO: do that automatically
    save: True
    checkpoint: True
    checkpoint_filepath: "/scratch/project_465000454/tmp/%DEFAULT.EXPID%/"
    out_filepath: "/scratch/project_465000454/tmp/%DEFAULT.EXPID%/"

```
</Details>

Note, for TEMP (radiosounding) and AMSU-A (satellite) Parts - modeled data at both `sfc` and `pl` levels are simulteniously required (such joint request is in development by BSC) 

**How to run:** runscript `run_obsall.py` is used to execute OBSALL application:
<Details>

```
#!/scratch/project_465000454/devaraju/SW/LUMI-23.03/C/python-climatedt/bin/python
# OBSALL Apps (3 parts: SYNOP, TEMP, AMSU-A observations)

# Import required libraries
import sys
import subprocess

# IN IMPLEMENTATION
# --- Processing ground-based observations (SYNOP)
print('**********************************************************')
print('DestinE Climate Digital Twin - OBSALL Apps')
print('--- Processing ground-based observations (SYNOP)')
print('**********************************************************')
command_synop_run = "cd SYNOP; pwd; ./main_synop.sh; exit 0"
subprocess.run(command_synop_run, shell=True, check=True, executable="/bin/bash")

# IN DEVELOPMENT
# --- Processing radiosounding observations (TEMP)
#command_radsound_run = "cd RADSOUND; pwd; ./main_radsound.sh; exit 0"
#subprocess.run(command_radsound_run, shell=True, executable="/bin/bash")

# IN DEVELOPMENT
# Processing satellite observations (AMSU-A)
#command_satellite_run = "cd SATELLITE; pwd; ./main_satellite.sh; exit 0"
#subprocess.run(command_satellite_run, shell=True, executable="/bin/bash")

sys.exit(0)

```
</Details>

... (coming soon) ...

# Disclaimer
The OBSALL package is in a developement phase by the University of Helsinki team, led by Heikki Järvinen (heikki.j.jarvinen@helsinki.fi). The team includes also Jouni Räisänen (jouni.raisanen@helsinki.fi), Lauri Tuppi (lauri.tuppi@helsinki.fi), Madeleine Ekblom (madeleine.ekblom@helsinki.fi), and Alexander Mahura (alexander.mahura@helsinki.fi). Some features are still not implemented and you may expect bugs. For feedback and issue reporting, feel free to open an issue in: https://earth.bsc.es/gitlab/digital-twins/de_340/obsall/-/issues

# Support
For all feedback, comments, and issues, feel free to open a relevant issue or email alexander.mahura@helsinki.fi

# Citing OBSALL package
If you use the OBSALL package in your research or publications, please cite using the following format:
<Details>

```
@software{obsall,
  author       = {Heikki Järvinen, Jouni Räisänen, Lauri Tuppi, Madeleine Ekblom, Alexander Mahura},
  title        = {ObsAll: Processing ground-based, radiosounding and satellite observational data for streamed climate modeled data},
  year         = {2023},
  publisher    = {GitLab},
  journal      = {Barcelona Supercomputing Center, Earth Sciences GitLab},
  howpublished = {\url{https://earth.bsc.es/gitlab/digital-twins/de_340/obsall/newbranch}},
}
```
</Details>

# Acknowledgments
We would like to acknowledge many contributors who have helped develop, test, and maintain this package, along with their highly valued advice.
Development of OBSALL package is supported by European Union Contract DE_340_CSC - Destination Earth Programme Climate Adaptation Digital Twin (Climate DT).
