Overview

The main objective of the C3S_512 contract Work Package (WP) 1 is to assess the quality of the Copernicus Climate Change Service (C3S) Climate Data Store (CDS) datasets, setting the minimum requirements and baseline criteria for including new products in the catalogue and providing comprehensive Evaluation and Quality Control (EQC) information and guidance for users of the dataset products. For this purpose, a data quality check should be performed to ensure that data ingested and provided by the CDS responds to the data models defined for each dataset together with the correct values. C3S_512 has developed a data quality checker based on the existing well-known scientific libraries, including current C3S convention checker, and other tools from the climate community. Due to the variety and heterogeneity of potential datasets and data models to be checked, the data quality checker (DQC) has been focusing on datasets from the multi-model seasonal forecasts and the reanalysis.

development environment

How to set up the development environment either to run from the VM or the workstation (work in progress)

architecture

High level description of the architecture of the C3SDatachecker(work in progress)

user_guide

how to run the checker, outputs generated (work in progress)

Setup development environment

add to your .bashrc file the following lines:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/shared/earth/software/Miniconda3/4.7.12-foss-2015a/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/shared/earth/software/Miniconda3/4.7.12-foss-2015a/etc/profile.d/conda.sh" ]; then
        . "/shared/earth/software/Miniconda3/4.7.12-foss-2015a/etc/profile.d/conda.sh"
    else
        export PATH="/shared/earth/software/Miniconda3/4.7.12-foss-2015a/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

and then you should execute the following line

export PATH="/data/miniconda3/bin:$PATH"

then you should be able to run a quick test in the VM:

conda activate /shared/software/envs/dqc
cd /shared/software/c3s512-wp1-datachecker/dqc_wrapper
python checker.py <config_file>

Folders in the VM

there are two main partitions, /data and /shared some of the most important directories to know about are the following:

- checker running scripts
/shared/QAR_dqc/bin

- location for downloading of data
/shared/cds_downloads/

- Log files of the ongoing downloads
/shared/cds_downloads/logs/

- location for the results of the checks (QARs)
/shared/QAR_dqc/results

- location for results after the process of renaming
/shared/QAR_dqc/results_cms

- current checker installations
/data/joan/c3s512-wp1-datachecker
/shared/software/c3s512-wp1-datachecker

Current checker installations points/use these folders