# Python AMIP reader <!-- omit in toc -->

Conversion from Fortran to Python of the EC Earth 4 project AMIP reader using the OASIS API from pyOASIS. This repository contains tests for running both implementations (Fortran code is taken from [EC Earth v3.3.3.1](https://earth.bsc.es/gitlab/es/auto-ecearth3/-/tree/3.3.3.1)) using the current setup for IFS and AMIP from EC Earth v3.3.3.1.

Running an example consists on executing one of the implementations (Fortran or Python) along with a mock component acting as IFS. This mock component simulates IFS reception behavior as designed in EC Earth v3.3.3.1. The detailed specification can be found below, with the coupling interval and run length taken directly from the `namelist.amip` file.
```python
# Constants
NAMELIST_FILE_NAME = 'namelist.amip'
L080_NX = 35718 # number of LAND grid cells at T159 resolution
L128_NX = 88838  # number of LAND grid cells at T255 resolution
L256_NX = 348528 # number of LAND grid cells at T511 resolution
```

## Table of contents <!-- omit in toc -->
- [Run an example](#run-an-example)
  - [Requirements](#requirements)
    - [Python dependencies](#python-dependencies)
  - [Compile](#compile)
  - [Run](#run)
- [Run the tests](#run-the-tests)
- [Visualize netCDF output files](#visualize-netcdf-output-files)
  - [Generate GIF files](#generate-gif-files)


## Run an example

### Requirements
This documentation assumes you are running an UNIX-based OS with Python3 in the PATH, understand the basics of OASIS and are able to compile and run pyOASIS.

#### Python dependencies
The Python implementation relies on the Python modules `f90nml`, `netCDF4` and `numpy` for running and the command `cdo` for testing. In order to install these dependencies, you may execute the following code.
```bash
pip3 install f90nml netCDF4 numpy
apt install -y cdo
```
Or if running (and testing) in MN4:
```bash
module load python/3.6.1
module load netcdf/4.2
module load cdo/1.9.3
```

Due to the large size of some of the required files and datasets, they are not provided in this repository. It is necessary to download them from MN4 before executing any code. Use the following script to download the required files (note you must change `bsc32074` to your username in MN4):
```bash
scp -r bsc32074@mn1.bsc.es:/gpfs/projects/bsc32/models/ecearth/v3.3.3.1/inidata/oasis/AMIP/* ./tests/data
scp -r bsc32074@mn1.bsc.es:/gpfs/projects/bsc32/models/ecearth/v3.3.3.1/inidata/amip-forcing/* ./tests/data/forcing
scp -r bsc32074@mn1.bsc.es:/gpfs/projects/bsc32/models/ecearth/v3.2.1/inidata/oasis/AMIP-reader/OLD_AMIP_NL/* ./tests/data/primavera
scp -r bsc32074@mn1.bsc.es:/gpfs/projects/bsc32/models/ecearth/v3.2.2/inidata/amip-primavera/* ./tests/data/primavera
```
If you are downloading the data from PRIMAVERA (+74 GB), it is highly recommended to do it in the background:
```bash
nohup scp -r bsc32074@mn1.bsc.es:/gpfs/projects/bsc32/models/ecearth/v3.2.2/inidata/amip-primavera/* ./tests/data/primavera > nohup.out 2>&1
# ctrl + z
bg
```

Or if running in MN4:
```bash
cp /gpfs/projects/bsc32/models/ecearth/v3.3.3.1/inidata/oasis/AMIP/* ./tests/data/forcing
cp /gpfs/projects/bsc32/models/ecearth/v3.3.3.1/inidata/amip-forcing/* ./tests/data/forcing
cp /gpfs/projects/bsc32/models/ecearth/v3.2.1/inidata/oasis/AMIP-reader/OLD_AMIP_NL/* ./tests/data/primavera
cp /gpfs/projects/bsc32/models/ecearth/v3.2.2/inidata/amip-primavera/* ./tests/data/primavera
```

### Compile
Each of the example codes is located in its respect folder inside the [`./sources/amip-forcing/`](./sources/amip-forcing/) folder ([`src/fortran/`](./sources/amip-forcing/src/fortran/) and [`src/python/`](./sources/amip-forcing/src/python/)). In the source folder for the Fortran code there is a [Makefile](./sources/amip-forcing/src/fortran/Makefile). Before comopiling OASIS, you must create a `make.inc` file which redirects to a make file with your configuration. You may issue the following command to create your own `make.inc` file:
```bash
cp sources/oasis3-mct/util/make_dir/make.templ.inc sources/oasis3-mct/util/make_dir/make.inc 
```

This would be the contents in `make.inc`:
```
curr_path = $(dir $(realpath $(lastword $(MAKEFILE_LIST))))
include  $(curr_path)/make.linux_gnu_openmpi4
```
 
For running in MN4, you may change the `include` path to `make.mn4`, so the file contents of `make.inc` would look like this:
```
curr_path = $(dir $(realpath $(lastword $(MAKEFILE_LIST))))
include  $(curr_path)/make.mn4
```

By executing the following command, you will compile OASIS, pyOASIS and the Fortran implementation of this model.
```bash
(cd ./sources/amip-forcing/src/fortran && make clean)
(cd ./sources/amip-forcing/src/fortran && make oasis)
(cd ./sources/amip-forcing/src/fortran && make)
```

If this is the first time you are compiling pyOASIS in the current session, you must update `LD_LIBRARY_PATH` and `PYTHONPATH` environment variables. Luckily, pyOASIS provides a automatic script:
```bash
source sources/oasis3-mct/generated/python/init.sh
```
```bash
source sources/oasis3-mct/generated/python/init.csh
```
Alternatively, these lines could be added to your `~/.bashrc` or `~/.cshrc` file.

### Run
In order to run the example code with your desired programming language you must execute the script `run_example.sh` inside the [`tests/`](./tests/) folder. Here is an example on how to run the example with the AMIP reader running in Python and the IFS mock component with 4 MPI processes from 1990-01-01 to 1991-01-01 without a fixed year.
```bash
bash ./run_example.sh python forcing 4 1990-01-01 1991-01-01 0 L128 true
```

Here is the complete specification for the script:
```bash
./run_example.sh <lang> <amip_mode> <ifs_nproc> <leg_start_date> <leg_end_date> <ifs_cmip_fixyear> <ifs_grid> <amip_interpolate>
```

Running this script will create a new folder in the root folder of this repository called `work_<lang>_<amip_mode>_<ifs_nproc>_<leg_start_date>_<leg_end_date>_<ifs_cmip_fixyear>_<ifs_grid>_<amip_interpolate>` with all the output and debug information from the run.

## Run the tests
Execute the following command in the root folder of this repository to run all tests **except the large test** (make sure to meet all the [requirements](#requirements) stated above).

If you are on MN4, it is highly recommended to start first an interactive session:
```
salloc -p interactive
```

```bash
bash run_tests.sh
```
If you also want to run the large test along with the other ones, use the following:
```bash
bash run_tests.sh 1
```

This command will run both implementations (Fortran and Python) of the AMIP reader and check if all of them result in the same output. Here is an example of a successfull test run:
```
Running tests WITHOUT the large test. Use ./run_tests.sh 1 to run all tests including the large test.
*****************************************************************
Running tests...
*****************************************************************
Checking results...
Checking ./work_python_forcing_2_1892-01-01_1894-01-01_0_L128_true vs ./work_fortran_forcing_2_1892-01-01_1894-01-01_0_L128_true
cdo    diff: Processed 2 variables over 1462 timesteps [0.23s 52MB].
cdo    diff: Processed 2 variables over 1462 timesteps [0.22s 52MB].
cdo    diff: Processed 2 variables over 1462 timesteps [0.30s 52MB].
cdo    diff: Processed 2 variables over 1462 timesteps [0.31s 52MB].
...
Checking ./work_python_forcing_4_1990-01-01_1991-01-01_0_L128_true vs ./work_fortran_forcing_4_1990-01-01_1991-01-01_0_L128_true
cdo    diff: Processed 2 variables over 2192 timesteps [0.33s 52MB].
cdo    diff: Processed 2 variables over 2192 timesteps [0.33s 52MB].
cdo    diff: Processed 2 variables over 2192 timesteps [0.45s 53MB].
cdo    diff: Processed 2 variables over 2192 timesteps [0.46s 53MB].
Tests PASSED
```

## Visualize netCDF output files
In order to visualize the results from an experiment, you may first fix the grid and time axis of the EXPOUT generated files. First, move into the work directory created when running the experiment. For example:
```bash
cd ./tests/work_python_forcing_4_1990-01-01_1991-01-01_0_L128_true/
```

Then, from inside that directory, run the following script with the start date as the argument:
```bash
bash ./convert-ece4pyreader-grids.sh 19900101
```

As a result, you will now have 8 new files in the working directory for the experiment: `A_Ice_frac.nc`, `A_Ice_frac.grb`, `A_SST.nc`, `A_SST.grb`, `AMIP_sst.nc`, `AMIP_sic.nc`, `areas_AMIP.nc`, `areas_L128.grb`, `areas_L128.nc`, `masks_AMIP.nc`, `masks_L128.grb` and `masks_L128.nc`.

### Generate GIF files
Optionally, you may want to save visualization as a GIF file. In order to achieve this, you may first run `./convert-ece4pyreader-grids.sh` as before and then the script `nc_to_gif.py` located in the [`tests/`](./tests/) folder. If you are in the working folder and want to visualize the `A_Ice_frac.nc` file, you may run the following command:

```bash
./nc_to_gif.py work_python_forcing_4_1990-01-01_1991-01-01_0_L128_true A_Ice_frac
```

Here is the complete specification for the script:
```bash
./nc_to_gif.py <work_dir> <file>
```

Running the script will generate two files: a raw GIF file and an optimized one. In this case `A_Ice_frac.gif` and `A_Ice_frac_optimized.gif` are generated. `A_Ice_frac_optimized.gif` is displayed below:

![A_Ice_frac_optimized](https://earth.bsc.es/gitlab/es/ec-earth4/uploads/2cbefdd0b3f9ebb10ceca44832511729/A_Ice_frac_optimized.gif)