# EC-Earth Data-Coupler  <!-- omit in toc -->
**NOTE**: Older version of ReadMe is available in [`docs/README_old.md`](./docs/README_old.md)

## Introduction 
The Data Coupler is being developed at the BSC to implement a CO2 box model for EC-Earth3 and to read and interpolate CO2 anthropogenic emissions (from e.g. CMIP6 input4MIPS) and CO2 surface fluxes (from e.g. vegetation+fire+ocean fluxes from SiB4) for EC-Earth4-CC. 

## Getting the model 
The model can be cloned from [`BSC-ES gitlab repository`](https://earth.bsc.es/gitlab/es/python-amip-reader)  


```
git clone https://earth.bsc.es/gitlab/es/python-amip-reader.git
cd python-amip-reader
git checkout data-coupler
```


### Setting up the model
First copy the datasets from Dataset repositories  based on the machine. 

**on MN4 :** 
```
datadir=/gpfs/scratch/bsc32/bsc32013/data-coupler/data
```

**on HPC2020:**
```
datadir=/hpcperm/c3ap/data-coupler/data
```

and then copy them as follows: 

```
cd python-amip-reader
mkdir data
cp -r $datadir/* data/
```
or, if you don't want to duplicate data and will not add any new dataset, create a link to the main Dataset folder.
```
cd python-amip-reader
ln -s $datadir data
```

## Compile
Each of the example codes is located in its respect folder inside the [`./sources/amip-forcing/`](./sources/amip-forcing/) folder ([`src/fortran/`](./sources/amip-forcing/src/fortran/) and [`src/python/`](./sources/amip-forcing/src/python/)). In the source folder for the Fortran code there is a [Makefile](./sources/amip-forcing/src/fortran/Makefile). Before compiling OASIS, you must create a `make.inc` file which redirects to a make file with your configuration. You may issue the following command to create your own `make.inc` file:

```bash
cp sources/oasis3-mct/util/make_dir/make.templ.inc sources/oasis3-mct/util/make_dir/make.inc 
```

This would be the contents in `make.inc` if running on your own laptop:
```
curr_path = $(dir $(realpath $(lastword $(MAKEFILE_LIST))))
include  $(curr_path)/make.linux_gnu_openmpi4
```
 
For running in BSC Marenostrum4 (mn4), you may change the `include` path to `make.mn4`, so the file contents of `make.inc` would look like this:
```
curr_path = $(dir $(realpath $(lastword $(MAKEFILE_LIST))))
include  $(curr_path)/make.mn4
```

For running in ECMWF HPC2020 (hpc2020), you may change the `include` path to `make.hpc2020`, so the file contents of `make.inc` would look like this:
```
curr_path = $(dir $(realpath $(lastword $(MAKEFILE_LIST))))
include  $(curr_path)/make.hpc2020
```

**WARNING**: `make.mn4` and `make.hpc2020` assume the proper modules are loaded.

For `make.mn4` this is done with the following command:

```
module load impi/2017.4
module load mkl/2017.4
module load intel/2021.4
module load python/3.6.1
module load netcdf/4.2
module load cdo/1.9.3
```

For `make.hpc2020` the following:   

```
module reset ;
module load python3/3.10.10-01 
module load prgenv/intel 
module load intel/2021.4.0
module load hpcx-openmpi/2.9.0
module load hdf5-parallel/1.12.2
module load netcdf4-parallel/4.9.1
```


For **MN4** and **HPC2020** you can use the following snippet to build oasis 

```
cd $HPCPERM/work/python-amip-reader/sources/oasis3-mct/util/make_dir
cp make.templ.inc make.inc
# change this line
# include  $(curr_path)/make.mn4
# to
# include  $(curr_path)/make.hpc2020
 
cd $HPCPERM/work/python-amip-reader/sources/amip-forcing/src/fortran
 
module reset ; module load python3/3.10.10-01 prgenv/intel intel/2021.4.0 hpcx-openmpi/2.9.0 hdf5-parallel/1.12.2 netcdf4-parallel/4.9.1
 
make realclean && make oasis && make
```
If this is the first time you are compiling pyOASIS in the current session, you must update `LD_LIBRARY_PATH` and `PYTHONPATH` environment variables. Luckily, pyOASIS provides a automatic script:
```bash
source sources/oasis3-mct/generated/python/init.sh
```
```bash
source sources/oasis3-mct/generated/python/init.csh
```
Alternatively, these lines could be added to your `~/.bashrc` or `~/.cshrc` file.

For other platforms you may need to modify or add a new makefile for your specific MPI setup.For more information lease check the [`README_old.md`](./docs/README_old.md#compile)

## Running the model

To run the model you need to edit file  `$HPCPERM/work/python-amip-reader/scripts/launch-hpc2020.cmd` and make sure the proper configuration is un-commented at the end, for example


For a simple *AMIP* run using the amip-forcing :
```
bash ./run_example.sh datacoupler forcing 4 2005-01-01 2005-03-01 0 128 true false template_conf_datacoupler_forcing.yaml
```
For the *AMIP Forcing + co2flux* model :
```
bash ./run_example.sh datacoupler co2flux 4 2005-01-01 2005-03-01 0 128 true false template_conf_datacoupler_co2flux.yaml
```
For the *CO2BOX* model :
```
bash ./run_example.sh datacoupler co2box 1 2005-01-01 2005-03-01 0 128 true false template_conf_datacoupler_co2box.yaml
```

### YAML input structure 

1. **DataCoupler**  

Three YAML templates are available for DataCoupler  variations:
- [`template_conf_datacoupler_forcing.yaml`](./scripts/template_conf_datacoupler_forcing.yaml) a replacement of the EC-Earth3 amip-forcing which reads SST and SIC from input4MIPS and sends them to IFS
- [`template_conf_datacoupler_co2box.yaml`](./scripts/template_conf_datacoupler_co2box.yaml) : a global CO2 box model which reads CEDS CO2 surface concentrations and emissions from input4MIPS and sends a global, surface CO2 field to IFS
- [`template_conf_datacoupler_co2flux.yaml`](./scripts/template_conf_datacoupler_co2flux.yaml) : a basic CO2 flux coupler which reads CEDS CO2 emissions from input4MIPS and sends dummy land,ocean and anthropogenic CO2 fluxes to IFS

They include 3 sections *Model Information*, *Run Information* and *Coupling Information*, some of which should be filled by the users and some (mainly the *Run Information*) are filled by the [`run_example.sh`](./scripts/run_example.sh) script as it is executed.  The generated YAML input will be stored under `name DataCoupler_exp_conf.yaml` . The  *Coupling Information* section contains a list of files to read (`FileInputVars`, containing information on the netcdf file and variable to read and other details), and another list of oasis exchange fields between the Data Coupler and the IFS toy model (`OasisOutputVars`, containing details on the sending and receiving of the fields). At every timestep of duration `TimeStepSec` each FileInputVar is read and its content sent to the OasisOutputVar corresponding to its oasis_name field. For the given example below, the `AMIP_sst_monthly` variable is sent by the DataCoupler as the `AMIP_sst` oasis field, which is received by the IFS toy model as the `A_SST` field.

Below, you will find a copy of the [`template_conf_datacoupler_forcing`](./scripts/template_conf_datacoupler_forcing.yaml) config file. 

```
cat << EOF
# YAML input file for DataCoupler, It will be ingested and turn to Experiment YAML file 

## Model Information
ModelNameSend        : AMIPFORC
ModelNameReceive     : IFS_TOY
LogFileName          : amip.log

## Run Information  
RunLengthSec    : ${leg_length_sec}     
TimeStepSec     : ${cpl_freq_amip_sec}
StartYear       : ${leg_start_date_yyyymmdd:0:4}
StartMonth      : ${leg_start_date_yyyymmdd:4:2}
StartDay        : ${leg_start_date_yyyymmdd:6:2}
FixYear         : ${ifs_cmip_fixyear}
GridInfo        : 88838   # number of grid cells : L080_NX = 35718 @ T159 resolution / L128_NX = 88838  @ T255 resolution / L256_NX = 348528  @ T511 resolution

## Coupling Information  
FileInputVars:
   AMIP_sst_monthly :  { id : AMIP_sst_monthly,  grid_name: AMIP,     oasis_name: AMIP_sst,        file_pattern: tosbcs_input4MIPs_SSTsAndSeaIce_CMIP_PCMDI-AMIP-1-1-3_gn_187001-201706.nc,    netcdf_variable: tosbcs,        yref_min : 1870, yref_max: 2016, timestep:  monthly, interpolate : true,  scale_factor: 1,    offset: 273.15, min: 271.38, max: ,  update: true, accum: true}  
   AMIP_sic_monthly :  { id : AMIP_sic_monthly,  grid_name: AMIP,     oasis_name: AMIP_sic,        file_pattern: siconcbcs_input4MIPs_SSTsAndSeaIce_CMIP_PCMDI-AMIP-1-1-3_gn_187001-201706.nc, netcdf_variable: siconcbcs,     yref_min : 1870, yref_max: 2016, timestep:  monthly, interpolate : true,  scale_factor: 0.01, offset: 0,      min: 0,      max: 1, update: true, accum: true}

OasisOutputVars:
   AMIP_sst :       { send_id: AMIP_sst,       receive_id: A_SST,         send_grid_name: 'AMIP',      receive_grid_name: L128,     timestep: 'daily', scale_factor: 1, offset: 0, reset: true }
   AMIP_sic :       { send_id: AMIP_sic,       receive_id: A_Ice_frac,    send_grid_name: 'AMIP',      receive_grid_name: L128,     timestep: 'daily', scale_factor: 1, offset: 0, reset: true }

LDebug : false
EOF
```

2. **Toy Model**  

Three YAML templates are available for Toy Model scripts, which are used for the EC-Earth3 Toy Model:
[`ifs_toy_model.yaml`](./sources/ece3-toy-model/ifs_toy_model.yaml)
[`nemo_toy_model.yaml`](./sources/ece3-toy-model/nemo_toy_model.yaml)
[`runoff_toy_model.yaml`](./sources/ece3-toy-model/runoff_toy_model.yaml)

They include information about the  *model*, *simulation* and *coupling* for three components of the EC-Earth3 Toy model (IFS, NEMO and Run off), some which should be filled by the users. 

Below, you will find a copy of the [`ifs_toy_model.yaml`](./sources/ece3-toy-model/ifs_toy_model.yaml). 


```
# yaml metadata 

model:
  ModelName     : IFS_TOY

simulation:
  coupling_interval : 2700
  run_length_sec : 86400
  # Definition of the local partition grid cell 
  # L080_NX = 35718  # number of LAND grid cells at T159 resolution
  # L128_NX = 88838  # number of LAND grid cells at T255 resolution
  # L256_NX = 348528  # number of LAND grid cells at T511 resolution
  nx : 88838

coupling: 
  out_vars : [A_TauX_oce, A_TauY_oce, A_TauX_ice, A_TauY_ice, A_Qns_mix, A_Qs_mix, A_Evap_total, A_Precip_liquid, A_Precip_solid, A_Qs_ice, A_Qns_ice, A_dQns_dT, A_Evap_ice, A_Runoff, A_Calving]
  in_vars : [A_SST, A_Ice_temp, A_Ice_albedo, A_Ice_frac, A_Ice_thickness, A_Snow_thickness]
  restart_file : /path/
```

### Submitting runs 

There are 2 ways to run the model: 

1. run an **interactive session** and launch the job with bash (for short runs and debugging)

```
ecinteractive -c6 -t 12:00:00 -m 32GB
cd $HPCPERM/work/python-amip-reader/scripts/
bash launch_ece3_data_coupler_hpc2020.cmd
```

2. launch the job with **sbatch** (for longer runs)

```
cd $HPCPERM/work/python-amip-reader/scripts/
sbatch launch_ece3_data_coupler_hpc2020.cmd
```

If the run is successful, output will be present in a folder representing the run you made e.g. `work_fortran_forcing_4_2005-01-01_2005-03-01_0_128_true`

Most of the debug output files produced by oasis (containing the fields sent by the data-coupler and received by the ifs toy model) are post-processed to netcdf files which proper grid and time metadata using [`convert-ece4pyreader-grids.sh`](./scripts/convert-ece4pyreader-grids.sh). These files can be viewed with ncview and consulted with cdo or ncdump.


## Visualize netCDF output files
Please refer to [`README_old.md`](./docs/README_old.md#visualize-netcdf-output-files) for more information about how to visualize netCDF output files. 
