# OBS Apps
This repository contains the scripts related to the Observation Apps - components for ground-based (SYNOP), radiosounding (TEMP), and satellite (AMSU-A) obs data) of the Climate Adaptation Digital Twin (Climate DT). The development is done in the frameworks of the Destination Earth initiative.

# Description
The OBS application is currently being developed as a set of scripts (written in `bash`, `fortran`, `python`):
1. for pre-processing (with operators of `cdo`, Climate Data Operators, software) gsv_extracted modeled data (over latitude-longitude domain) for selected meteorological variables into hourly time-slices (00...23 UTCs);
2. for extracting (using `sql` with `odp_api` software) selected meteorological variables at hourly time-slices for observations from: (i) ground-based synoptical stations at fixed at the surface geographical locations/points (i.e., 2D: latitude, longitude); (ii) vertical radiosounding of the atmosphere at changing locations/points (3D: latitude, longitude, single pressure level), and (iii) satellite at changing locations/ points (3D: latitude, longitude, multiple pressure levels);
3. for extracting and interpolating (using `cdo` operators - for synop; & expecting/using `polytope` - for radiosounding and satellite) modeled data for the same time-slices for same selected meteorological variables into corresponding locations/points of (synop, radiosounding, satellite) observations, and adding (using `import` with `odb_api`) such data to ODB;
3. for calculating/ producing relevant statistics such as quantile rank histogram statistics and plots (t-test and others to be added)

# Part SYNOP (ground-based observations)
1. `main_synop.sh` - main bash-script to run Apps for observational data for synoptical stations.
2. `gsv_mod_data.sh` - bash-script to run pre-processing (using `cdo` operators) of modelled data extracted with gsv interface over global domain for selected meteorological variables (measured at synoptical station) into hourly time-slices and saving (temporary) to separate hourly nc-files.
3. `synop_obs.sh` - bash-script to run reading and extraction (using `sql` with `odb_api` software) of selected meteorological variables at hourly time-slices for observations from ground-based synoptical stations at fixed at the surface geographical (latitude, longitude) locations over selected geographical domain (or over the globe) and saving (temporary) to dat-file. Note, that list of available  meteorological variables in open multi-year data ODB includes: 91 - total amount of clouds; 108 - sea level pressure; 80 - 1-hour precipitation; 58 - relative humidity; 999 - 10-minute precipitation intensity; 71 - snow depth; 39 - 2m temperature; 40 - dew point temperature; 62 - visibility; 111 - wind direction; 261 - maximum wind gust in last 10 minutes; 221 -surface wind speed.
4. `synop_mod.sh` - bash-script to run reading, extraction and interpolation (using `cdo` operators) of modelled data for selected meteorological variables at the same hourly time-slices to geographical (latitude, longitude) locations of synoptical stations, and saving (temporary) to dat-file, and adding (using `import` with `odb_api`) such data to ODB. Interpolation is preceeded by constructing unstructured grid based on locations of synop stations and calculating weights for such grid. 
5. `graph_mod_obs.py` - python-script to run an internal self-control in calculating differences between observed and modelled data for locations of synoptical stations 
6. `synop_stats.sh` - bash-script to run (addtional `bash`, `fortran`, `python` scripts) calculating/producing:
- (i) rank histograms for all synop stations for 00, 06, 12 and 18 UTCs (with `produce_rank_histograms_all_stations.sh`, using `rank_histograms_one_station.f95` and `sql/import` with `odb_api`), 
- (ii) standard plots for each synop station (with `produce_standard_plots_all_stations.sh` using `plot_quantiles_rankhist.py` and  `sql` with `oadb_api`; see an example of such plot in **Figure 1**), and 
- (iii) summary rank histograms for all stations (`summary_rank_histograms_all_stations.sh` using `rank_histogram_summary_statistics.f95`, `plot_p_values_map.py`, `plot_rank_hist_sum_all_stations.py` and `sql` with `odb_api`; see examples of such plots in **Figure 2** and **Figure 3**). 
- Note, needed input includes: list of synop stations with geographical coordinates, modelled data at stations coordinates, pre-computed quantiles as a function of time of year and rank histogram bootstrap mean square deviation (MSD) values. 
 - ![Semantic description of image](FIGSEXAMPLES/destine_obs_synop_figs.jpg "Plots for 2 m air temperature ")* Plots for 2 metre air temperature (T2m) for period of 1 Jan - 31 Dec 2020: **Figure 1** - Quantiles/ time series data (00, 06, 12, and 18 UTCs) and rank histogram (combination of 00, 06, 12, and 18 UTCs) for synop station 126736; **Figure 2** - P-values for T2m quantiles; **Figure 3** - Rank histogram summary statistics: normalized p-value frequencies and normalized quantile frequencies*

# Part TEMP (radiosounding observations)
1. `main_radsound.sh` - main bash-script to run Apps for observational data for radiosounding.
2. `gsv_mod_data.sh` - bash-script to run pre-processing (using `cdo` operators) of modelled data extracted with gsv interface over global domain for air temperature at 2 metre and at 850 hPa pressure level into hourly time-slices and saving (temporary) to separate hourly nc-files.
3. `radsound_obs.sh` - bash-script to run reading and extraction (using `sql` with `odb_api` software) of above-mentioned air temperatures at hourly time-slices for radiosounding observations at surface and at altitude at geographical (latitude, longitude, pressure level) locations of radiosounding stations over selected geographical domain (or over the globe) and saving (temporary) to dat-file.
4. `radsound_mod.sh` - bash-script to run reading, extraction and interpolation (using polytope) of modelled data for air temperature at surface and at altitude at the same hourly time-slices to geographical (latitude, longitude, pressure level) locations of radiosounding stations, and saving (temporary) to dat-file, and adding (using `import` with `odb_api`) such data to ODB.
5. `graph_mod_obs.py` - python-script to run an internal self-control in calculating differences between observed and modelled data for locations of radiosounding stations
6. `radsound_stats.sh` - bash-script to run (addtional `bash`, `fortran`, `python` scripts) calculating/producing: ...

# Part AMSU-A (satellite observations) 
... (coming soon) ...

# Disclaimer
The OBS package is in a developement phase by the University of Helsinki team, led by Heikki Järvinen (heikki.j.jarvinen@helsinki.fi). The team includes also Jouni Räisänen (jouni.raisanen@helsinki.fi), Lauri Tuppi (lauri.tuppi@helsinki.fi), Madeleine Ekblom (madeleine.ekblom@helsinki.fi), and Alexander Mahura (alexander.mahura@helsinki.fi). Some features are still not implemented and you may expect bugs. For feedback and issue reporting, feel free to open an issue in: https://earth.bsc.es/gitlab/digital-twins/de_340/obs/-/issues

# Support
For all feedback, comments, and issues, feel free to open a relevant issue or email alexander.mahura@helsinki.fi

