Home

s2dverification wiki

Set of tools to assess the performance of a model through the computation of typical prediction scores against one or more observational datasets or reanalyses (a reanalysis being a physical extrapolation of observations that relies on the equations from a model, not a pure observational dataset).

Description

This package is coded in R.

It allows you to load a variable from one or more experiments together with the reference data at the corresponding dates for validation where the reference data can be any observational dataset or reanalysis or several of them. Once loaded the experimental and reference data are organised into two matrices with similar structures that can be used as input arguments to the other functions to compute anomalies, climatologies, any prediction score and finally plot or animate the results.

If you want to keep abreast of the latest releases, bugs and discussions on s2dverification, subscribe to its mailing list by sending an e-mail to s2dverification-request@bsc.es with the subject 'subscribe'.

How to cite

N. Manubens et al., “An R Package for Climate Forecast Verification,” Environmental Modelling and Software 2018 (under review).

Contact persons

Code developed by [Barcelona Supercomputing Centre] (https://www.bsc.es/) (BSC-CNS).

Maintainer:

nicolau.manubens@bsc.es

Developers:

Check the [CRAN page] (https://cran.r-project.org/web/packages/s2dverification/index.html) page for the full list of developers.

Requirements

This package has two system dependencies, CDO and R (>= 2.14.1), apart from the R packages ncdf4, GEOmap, geomapdata, maps, mapproj, abind, parallel and bigmemory but these are installed automatically with

[Short examples] (short_examples)

Tutorials

The following tutorial given at the SPECS general assembly 2014 shows the basic feature of s2dverification. All the examples are done with the data included in the package on CRAN, so it can be done outside of IC3. All the example are corrected.

Time estimation: 1hour. Tutorial 1

The second tutorial is an extract of the tutorial done to learn how to use s2dverification for the ICTP-WRCP extreme event summer school. It is longer and more complete than the first one, and it uses the ENSEMBLES seasonal forecast, hence it cannot be done outside of the IC3.

Time estimation: 3 hours. Tutorial 2

User guide

To use the R s2dverification functions, open an R session and load the package with the command

library(s2dverification)

or, if you want to avoid collision of s2dverification function names with function names of other R packages in use in your session, with the command

requireNamespace('s2dverification', quiet = TRUE)

If you use the second option, you must call the functions in the package as

s2dverification::FunctionName() instead of FunctionName()

You can check the list of functions in the package with the command

help(package = s2dverification)

After the package is loaded, you can check information of a function with the command

?FunctionName

You'll see a quick description of the aim of the function, the input arguments, the output variables and contact references in case of questions.

To learn how to use these functions you can start by giving a look at the examples in the documentation pages of each function. For more advanced examples of usage you can run the scripts under the directory 'doc' of the s2dverification package ('inst/doc' in the raw GIT repository). Few arguments need to be specified as explained in the headers inside:

  plot_timeseries.R
  plot_maps.R

Use a development version

If you want to use an old version or a version still in development of the s2dverification, download the repository folder to your computer and check-out to the branch that contains the version you want to use.

Set the following environment variables with the commands

R_LIBS="/usr/local/lib/R/site-library:/usr/lib/R/library:/usr/lib/R/site-library:/cfu/software/R/debian7"
 export R_LIBS
 _R_CHECK_RD_XREFS_=false
 export _R_CHECK_RD_XREFS_

Go to the repository directory and build the package tarball with the command

R CMD build /path/to/repository

this will create a .tar.gz file in the current directory (repository main folder) with the source files compressed, ready to install.

Install the package locally with the command

R CMD INSTALL -l /path/where/to/install /path/to/tarball.tar.gz

Then you can load the library in R with the command

library(s2dverification, lib.loc = '/path/where/to/install')

and it will be ready to use.

[Change log] (change_log)

[User manual] (user_manual)

The user manual can be downloaded from CRAN.

[Dissemination] (disemination)

Slides on s2dverification from the tools meeting held on 2014/11/03: SPECS

Overview slides and big data issues from BSC-UniCan meeting: BSC-UniCan

Slides about R, its popularty and tricks for performance, from CES training talks on 2015/05/19: CES training

Internal s2dverification update meetings:

[Development] (development)

Package structure

In the 's2dverification' project main folder you can see a few files that fulfil the standard R package structure. These files are called the source files. The structure is as follows:

Folder 'R': Contains .R files with the code of the functions in the package, one function per file. The name of the file must be the same as the function it contains ending with '.R'.
Folder 'man': Contains .Rd files with the documentation of the functions and of the package, one file per function and one file for the documentation of the whole package. The name of the file must be the same as the function, ending with '.Rd'. This file should contain a section named 'examples' than contains example code that calls the function being documented.
Folder 'inst': Contains documentation files, images, etc. that will be provided to the user after the installation of the package. Be careful with the weight of the files, they should be as light as possible. For example, the folder 'inst/doc' contains demo scripts that show how to use the package. When the user installs the package, they will be able to check the scripts in the folder '/path/to/package/doc'. This folder also contains the folder 'config' which contains two s2dverification configuration files, one used at IC3 and which is loaded by default, and a sample configuration file.
File 'DESCRIPTION': Contains general information about the package, dependencies, authorship, etc. and is looked up automatically in the package build stage.
File 'NAMESPACE': Specifies what variable names are used from outside the package and what variable names are provided to the outside.

How to develop

Download the repository folder to your computer ( git clone https://earth.bsc.es/gitlab/es/s2dverification.git <local_folder> ).

Using GitLab and according to the GIT branching and testing strategy introduced in 2014/11, every time you want to perform a new development, you must follow the steps in the image, explained in detail below.

Developer steps

0- Log in GitLab and go to the GitLab project main page

 If a red message apears at the top, you can close it

1- Open a new issue

- Issues -> + New Issue -> Submit new issue
 - Put title, description and label. Write @username to notify people
 - Click on "Submit new issue"
 - You can then see a list with all the issues. Each is assigned an #issue_number

2- Create a branch from 'master'

EITHER from GitLab
 - Commits -> Branches -> New branch
 - Put name of the new branch and the name of the branch you check off from, in this case 'master'
 - Click on "Create branch"
 OR from terminal
 - git fetch origin
 - git checkout master
 - git pull
 - git checkout -b branch_name

3- Perform feature/bugfix commits

- Open terminal
 (- git fetch)(only if created from GitLab)
 - Checkout to the new branch
 - Make changes
 - Last commit must finish with 'Fixes #issue_number'

4- Check that everything is fine

- Run the script /shared/earth/software/scripts/test-s2dverification branch_name. It will do atomatically:
   a) rebase master into the development branch
   b) build the package tarball
   c) check the tarball
 - If any error raises during rebase do:
   git merge --no-ff master
   fix conflicts
   commit
   try 4- again
 - If any error raises during build or check:
   build and check manually (follow instructions in wiki in section How to develop > How to build and check manually)
   fix errors
   commit
   try 4- again
 - git push origin HEAD

5- Create a merge request

- Merge Requests -> New Merge Request
 - Choose origin and destiny
 - Click on "Compare branches"
 - Fill in title and description.
   Description field should contain testers referenced with @tester_name
   If any comment is needed put there
   Select a milestone for the feature, so that it's associated to a deadline. If there's no associated milestone you can create one or specify a deadline for the feature to be tested in the text
 - Assign to the project coordinator
 - Click on "Submit merge request"

Tester steps

6+7-

 · Run /shared/earth/software/scripts/test-s2dverification branch_name
 · If you run it from within the s2dverification GIT repository, the clone will be skipped
 · If any error raises please tell the developer
 · An R session will open where you can load the new version of the package with library(name_of_the_package)
 · If you want to test from Rstudio you can load an installation that the test script has generated (see script notifications)

8- When the tests are performed and passed approve the MR with an approval comment (write 👍 in a message in the MR discussion)

Coordinator steps

9- Validate and perform the merge clicking on “Accept Merge Request”, with “Remove source-branch” marked.

- It is important before accepting the merge request to run the test script on the branch which is going to be merged.
 - Even at that stage conflicts could appear. In that case, solve them meeting the developers involved.
 - After merging and removing a branch from GitLab, carefully delete the branch from your local Git repository with git branch -D branch_name

Steps to build and check a package manually

Set the following environment variables with the commands

_R_CHECK_RD_XREFS_=false
 export _R_CHECK_RD_XREFS_

You can place the commands above in your .bashrc file if you develop in this package frequently.

Go to the repository directory and build the package tarball with the command

R CMD build --resave-data /path/to/repository

This will create a .tar.gz file in the current directory (repository main folder) with the source files compressed, ready to install.

Check that the tarball is installable with the command

R CMD check --no-examples /path/to/tarball.tar.gz

This will advice you if there is any problem with the added changes and will run the example code in the '.Rd' files. Also will generate in your current directory the folder 's2dverification.Rcheck', that contains the log files of a set of tests to ensure the tarball will be installable. This folder also contains the folder 's2dverification' with an installation of the package and the file 's2dverification-manual.pdf', a handsome user manual generated automatically from the files in the folder 'man'. A copy of it is being kept in the project main folder.

If you wish, for tidy purposes, you can run the two previous commands in the project main folder. The tarball and the .Rcheck folder will be generated there breaking the package structure, but will be ignored when you push the changes to the repository.

If you wish, you can enable the execution of the example code removing the parameter '–no-examples':

R CMD check /path/to/tarball.tar.gz

The example execution takes a long time, but it is run at least before every release to CRAN.

Optionally you can start R and load the installation of the package in the 's2dverification.Rcheck' repository with the command

library(s2dverification, lib.loc = '/path/to/s2dverification.Rcheck/')

and check that the modified functionalities work properly again.

Steps to add a new function to a package

Put the code of the function in a file named 'FunctionName.R' and place it in the R folder.
Make sure the code of the function follows the style rules (information below header, history, …).
Load the function with source() in an R session.
Type prompt(FunctionName). This will generate an .Rd file in the working directory.
Place the .Rd file into 'man' folder.
Fill in the .Rd file with the function documentation. Most of the text in this file should coincide with the text in the comments below the function header.

Reminder

Each function in the folder R has a corresponding .Rd file in the man folder, and the text in the .Rd will be used to generate the user manual and will appear when you type ?FunctionName.

Maintenance

Branch naming conventions

Main branch: master

Development branches: develop-feature or feature

Bugfix branches: develop-bugfixes-2.3.1 (when last release was 2.3.1)

Docfix branches: develop-docfixes-2.3.1 (when last release was 2.3.1)

Hotfix branches: develop-hotfixes-2.3.1-a (when last release was 2.3.1)

Upgrade branches: develop-upgrade-to-2.3.2 (when last release was 2.3.1)

Version numbers will be bumped as described in the Semantic Versioning conventions. See http://semver.org.

Steps to release

1- Check all open issues to see if some can be solved easily before releasing.

2- Open MR for all open bugfix branches.

3- Open MR for all open docfix branches.

4- Open MR for all open hotfix branches.

5- Close all MRs possible. For each:

- run test script before accepting
- ls -la everywhere to check no trash/huge files leak into master. Update .gitignore and - Rbuildignore if needed
- accept (removing source branch)
- git checkout master
- git fetch origin
- git pull

6- Run test script. Then run manually R CMD check without parameter –-no-examples and with parameter -–as-cran if releasing on CRAN. Try to include in test scripts all bugs detected since last release.

7- If small errors appear, fix in for example develop-hotfixes-2.3.1-b and go to 3-.

8- Open upgrade branch and open MR and merge as in 4-. Update at least:

- DESCRIPTION (date, version, dew depends, new authors)
- NAMESPACE (new depends)
- man/package-s2dverification.Rd
- pdf

9- Submit on CRAN.

10- Create a tag (if from GitLAB, do a 'git fetch origin' after).

11- Add any special messages in R code or add log-registering commands.

12- Build and install in /cfu/software, and in amdahl and in moore for each module possible.

13- Send release e-mail to s2dverification list, with all news since previous release.

14- Update change log on wiki page.

Style Guide

You can check the style guide for the s2dverification package here