# Hands-on 1.4: Provenance generation when running with autosubmit

Author: Albert Puiggros Figueras

## Goal

Create a SUNSET recipe to generate a set of JSON files containing the provenance information of the workflow execution combining autosubmit rocrate structure with the METACLIP ontologies. For this, we compute some skill metrics and scorecards plots with SUNSET, using Autosubmit to dispatch jobs in parallel. In the recipe, we request 3 start dates for January, Feburary and March (0101, 0201,0301). SUNSET will split the recipe into 3 atomic recipes, and Autosubmit will run 3 jobs, which process the verification, for each recipe in parallel. When all the scorecards are generted, the transfer_provenance job will be triggered transfering the SUNSET-generated provenance files to the autosubmits experiment folder to, lastly, create an rocrate object where the entire process description is encapsulated. 

## Why provenance?

Creating an RO-Crate object for your workflow is highly beneficial for reproducibility, transparency, and long-term storage. It encapsulates the entire process, including data, scripts, execution steps, and provenance information, ensuring others can easily reproduce and understand the results. By adhering to open standards, RO-Crates are interoperable with various tools, making data sharing and integration seamless. In this case, they store rich metadata, such as Autosubmit configuration files and Autosubmit job structures, which enhance discoverability and reuse. Fruther, it includes a set of JSON files that describe with detail the SUNSET worflow executed (one JSON file per job). For publications, RO-Crates provide a citable, machine-readable package of the workflow, ensuring durability and accessibility for future research or collaborative efforts.

## 0. Cloning the SUNSET repository (branch Dev-Provenance)

If you're completely new to SUNSET, the first step is to create a copy of the tool in your local environment. 
Open a terminal and `cd` to the directory where you would like to store your local copy of SUNSET. For example: `/esarchive/scratch/<your_username>/git/`. If a directory does not exist yet, you can create it with the `mkdir` shell command.

```shell
# Clone the GitLab repository to create a local copy of the code
git clone https://earth.bsc.es/gitlab/es/sunset.git

```
You should see a git folder "sunset" under the current directory. Now you have all the code, recipes, and scripts for running SUNSET.

## 1. Create Autosubmit experiment
Since we're going to use Autosubmit to dispatch jobs, we need to have an Autosubmit experiment. Note that SUNSET uses Autosubmit >= 4.1.11

```shell
module load autosubmit/4.1.11-foss-2021b-Python-3.9.6 
autosubmit expid -H mn5 -d "SUNSET use case 1_4" 
```
You will see the messages like below:

```shell
Autosubmit is running with 4.1.11
The new experiment "a83y" has been registered.
Generating folder structure...
Experiment folder: /esarchive/autosubmit/a81u
Generating config files...
Experiment a83y created
```
Note the experiment ID (in this snippet above a83y) down. We need it for the recipe later.

## 2. Modifying the recipe

The template recipe for this use case can be found in [ex1_4-recipe.yml](use_cases/ex1_4_autosubmit_provenance/ex1_4-recipe.yml). 
You should at least edit some items in the "Run" section:
- `output_dir`: The directory you want to save the outputs and logs
- `code_dir`: The directory where your SUNSET code is stored (i.e., the git folder)
- `auto_conf$script`: The path to the script ex1_2-recipe.yml
- `auto_conf$expid`: The experiment "xxxx" you just created
- `auto_conf$hpc_user`: You user ID on Nord3, which should be bsc032xxx
- `auto_conf$email_address`: Your email. You can also adjust other email notification parts up to your preference.

In the recipe, we ask for anomaly calculation after loading the data, calculate the skill scores and save the result for scorecards. In the Scorecard section, three regions are requested.

Note the use of parameter `Provenance` to request the provenance generation. Currently, provenance is available for the Loading and Aggregation. Further modules will have this provenance feature in the future, aiming to cover the entirely SUNSET workflows. Further, you have to include your personal orcid. If doesn't dispose of one yo can create one here [https://orcid.org/](https://orcid.org/): 

```shell
orcid: https://orcid.org/0009-0006-0707-6448
```
Turn the notifications off due to a current bugg in autosubmit.

```shell
 email_notifications: no
```

Issue: [Issue 1468](https://earth.bsc.es/gitlab/es/autosubmit/-/issues/1468)

## 3. The user-defined script

We need to have a script to define the modules to use and the steps of the workflow. Note that the script is for data loading and verification parts. The Scorecards module doesn't need to be included in this script. The Statistics module is called in order to compute some statistics that are needed to correctly to the ensemble correlation aggregation for the Scorecards.

The prepare_outputs() function is already incorporated into the launcher script (see the next section for details about launcher), so it does not need to be included in the user-defined script in this case. 
In its place, we will use the function read_atomic_recipe(). The recipe path is passed as an argument onto the R script. Below is an example of what the script should look like. 

Nothing need to be specified in teh script regarding provenance operations. Only with the recipe specifications is enough. 

NOTE: Don't try to copy and paste this code on an R terminal; it will not work without properly pre-processing the recipe and creating the necessary output folders first.

The beginning of the user-defined script should look like this:

```R
# Load modules
source("modules/Loading/Loading.R")
source("modules/Anomalies/Anomalies.R")
source("modules/Skill/Skill.R")
source("modules/Statistics/Statistics.R")
source("modules/Scorecards/Scorecards_calculations.R")
source("modules/Saving/Saving.R")

# Read recipe 
## (leave this part as-is! Autosubmit will automatically pass the atomic recipe as a parameter)
args = commandArgs(trailingOnly = TRUE)
recipe_file <- args[1]
recipe <- read_atomic_recipe(recipe_file)
```

The rest of the user-defined script can be written in the same way as any other SUNSET script. We load the data, calculate the anomalies, then compute the skill metrics and statistics, and we call `Scorecards_calculations()` to do some specific computations and save the result as netCDF files for Scorecards.

```R
# Load data
data <- Loading(recipe)
# Compute tos anomalies
data <- Anomalies(recipe, data)
# Compute skill metrics
skill_metrics <- Skill(recipe, data)
# Compute statistics
statistics <- Statistics(recipe, data)
# Pre-computations required for the Scorecards
Scorecards_calculations(recipe, data = data, 
                        skill_metrics = skill_metrics,
                        statistics = statistics)

```
Check the example script at [ex1_4-script.yml](use_cases/ex1_2_autosubmit_scorecards/ex1_2-script.R).
You can execute it as-is or copy it and modify it according to your needs.

## 4. Launch jobs and Use Autosubmit

We will start the jobs with the launcher. The SUNSET Launcher is a bash script named launch_SUNSET.sh that can be found in the main directory of the SUNSET repository. It runs in two steps:

1. Run the recipe checks, split the recipe into atomic recipes and create the directory for the outputs.
2. Modify the Autosubmit configuration of your experiment according to the parameters in the recipe.

The bash script needs two inputs: (1) [recipe](#2-modifying-the-recipe) (2) [R script](#3-the-user-defined-script).

 On your workstation or Nord3 under the SUNSET code directory, run:

```shell
bash launch_SUNSET.sh use_cases/ex1_4_provenance_autosubmit/ex1_4-recipe.yml use_cases/ex1_2_provenance_autosubmit/ex1_4-script.R 
```
You will see the messages similar to below:
```shell
[1] "Saving all outputs to:"
[1] "/esarchive/scratch/apuiggro/sunset/outputsex1_4-recipe_20241203113236"
INFO  [2024-12-03 11:32:36] SUNSET: SUbseasoNal to decadal climate forecast post-processing and asSEssmenT suite
    Copyright (C) 2024 BSC-CNS
    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.
    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.
    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.
INFO  [2024-12-03 11:32:38] Checking recipe: use_cases/ex1_4_provenance_autosubmit/ex1_4-recipe.yml
WARN  [2024-12-03 11:32:38] The element 'fcst_year' is not defined in the recipe. No forecast year will be used.
INFO  [2024-12-03 11:32:38] ##### RECIPE CHECK SUCCESSFULL #####
INFO  [2024-12-03 11:32:39] Splitting recipe in single verifications.
INFO  [2024-12-03 11:32:39] The main recipe has been divided into 3 single model atomic recipes, plus 0 multi-model atomic recipes.
INFO  [2024-12-03 11:32:39] Check output directory /esarchive/scratch/apuiggro/sunset/outputs/ex1_4-recipe_20241203113236/logs/recipes/ to see all the individual atomic recipes.
INFO  [2024-12-03 11:32:40] ##### AUTOSUBMIT CONFIGURATION WRITTEN FOR a83y #####
INFO  [2024-12-03 11:32:40] You can check your experiment configuration at: /esarchive/autosubmit/a83y/conf/
INFO  [2024-12-03 11:32:40] Please SSH into bsceeshub01 or bsceshub02 and run the following commands:
INFO  [2024-12-03 11:32:40] module load autosubmit/4.1.11-foss-2021b-Python-3.9.6
INFO  [2024-12-03 11:32:40] autosubmit create a83y
INFO  [2024-12-03 11:32:40] autosubmit refresh a83y
INFO  [2024-12-03 11:32:40] nohup autosubmit run a83y & disown
INFO  [2024-12-03 11:32:40] To generate the rocrate object (zip file) run the following command
          after the execution is finished:
INFO  [2024-12-03 11:32:40] autosubmit archive --rocrate a83y
INFO  [2024-12-03 11:32:40] The zip file will be saved in /esarchive/autosubmit/2024
```
You can see some useful information, like the the path to atomic recipes, the Autosubmit configuration files, and most importantly, follow the last lines to launch your experiment. 

```shell
ssh bsceshub01.bsc.es
(enter Hub machine)
module load autosubmit/4.1.11-foss-2021b-Python-3.9.6
autosubmit create a83y
autosubmit refresh a83y
nohup autosubmit run a83y & disown
```

Then, you can go to [Autosubmit GUI](https://earth.bsc.es/autosubmitapp/) to check the experiment status.

![Autosubmit graph.](Figures/autosubmit_graph.png)

As you can see, the Scorecards job is dependent on the Verification jobs. Once the 3 verification jobs are finished, the Scorecards job will start, followed by the transfer jobs.


# 5. Archive the autosubmit experiment and create rocrate object

After the experiment has executed correctly you can archive it requesting rocrate :

```shell
autosubmit archive --rocrate a83y -v

```

This will generate the zip file containing all the provenance metadata and the corresponding files. At the time, autosubmit stores the zip in a folder: 

```shell
/esarchive/autosubmit/2024

```

The output consists on a folder with a top file [ro-crate-metadata.json](Files/ro-crate-metadata.json) that contains all the rocrate provenance.

```shell
.
|- ro-crate-metadata.json
|- conf
|  |- autosubmit_a83y.yml
|  |- expdef_a83y.yml
|  |- jobs_a83y.yml
|  ...
|- status
|  |- a83y_20241122_1047.txt
|  |- a83y_20241122_1140.txt
|- tmp
   |- ex1_4-recipe_20241203113236
      |- logs
        |- recipes
          |- atomic_recipe_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0101.yml
          |- atomic_recipe_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0201.yml
          |- atomic_recipe_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0301.yml
          |- ex1_4-recipe.yml
      |- outputs
        |- provenance
          |- main_provenance_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0101.json
          |- main_provenance_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0201.json
          |- main_provenance_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0301.json
   ...
```
This is the structure of [ro-crate-metadata.json](Files/ro-crate-metadata.json): 

```shell
{
  "@context": "https://w3id.org/ro/crate/1.1/context",
  "@graph": [
    {
      "@id": "./",
      "@type": "Dataset",
      "conformsTo": [
        { "@id": "https://w3id.org/ro/wfrun/process/0.1" },
        { "@id": "https://w3id.org/ro/wfrun/workflow/0.1" },
        { "@id": "https://w3id.org/workflowhub/workflow-ro-crate/1.0" }
      ],
      "creator": { "@id": "https://orcid.org/0009-0006-0707-6448" },
      "datePublished": "2024-11-29T12:01:02+00:00",
      "description": "Test multiprov",
      "hasPart": [
        (...)
      ],
      "license": "Apache-2.0",
      "mainEntity": { "@id": "conf/metadata/experiment_data.yml" },
      "mentions": [
        { "@id": "#create-action" }
      ]
    },

    (...) 

    {
      "@id": "#metaclip-create-action",
      "@type": "CreateAction",
      "description": "SUNSET provenance based on METACLIP",
      "endTime": "2024-11-29T10:18:00Z",
      "instrument": {
        "@id": "file:///esarchive/scratch/apuiggro/sunset"
      },
      "isPartOf": {
        "@id": "#create-action"
      },
      "object": [
        { "@id": "atomic_recipe_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0101.yml" },
        { "@id": "atomic_recipe_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0201.yml" },
        { "@id": "atomic_recipe_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0301.yml" }
      ],
      "result": [
        { "@id": "/tmp/ex1_4-recipe_20241129101842/outputs/provenance/main_provenance_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0101.json" },
        { "@id": "/tmp/ex1_4-recipe_20241129101842/outputs/provenance/main_provenance_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0201.json" },
        { "@id": "/tmp/ex1_4-recipe_20241129101842/outputs/provenance/main_provenance_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0301.json" }
      ]
    }
  ]
}
```

Each json file in `tmp/ex1_4-recipe_20241203113236/outputs/provenance` contains the provenance of each individual job (as an atomic execution of a SUNSET workflow) run with autosubmit. When using the METACLIP intepreter [METACLIP Intepreter](https://metaclip.org/) it can be interactively visualized. Each node represents an entity (dataset, reggriding operation, command call, etc) and its related to other nodes with edges. Each edge represents a relation between entities. Entities can be further expanded to obtain more specific metadata related to the METACLIP definitions or further entity properties established by SUNSET. 

Here is an example: 

![METACLIP intepreter visualization of `main_provenance_sys-ECMWF-SEAS51_ref-ERA5_var-tas_reg-global_sdate-0201`.](Figures/metaclip_graph.png)

Combined with METACLIP, the Ro-Crate-defined provenance provides a nearly complete representation of the entire workflow execution. However, note that provenance is currently unavailable for the Skill and Anomalies modules. 

This same use case can be executed with nord3v2 machine instead of mn5 olny by modifying the values of 
`recipe$Run$filesystem` to esarchive and `recipe$Run$auto_conf$platform`to nord3v2. With this setting, the workflow structure looks like: 

![Autosubmit graph nord3v2.](Figures/autosubmit_graph2.png)