User Tools

Site Tools


tools:run_1st_exp

This is an old revision of the document!


Running an experiment at BSC

The idea of this documentation is to provide information for people running climate models at BSC. The goal is not to replace the autosubmit documentation, it is to provide a more local information, in particular to help new people in the climate prediction group to run EC-Earth simulations.

Please, feel free to fill and/or correct this documentation
Documentation to be checked and completed (Martin Ménégoz; 31/05/2016). It should be good that people include in this page some synthesised information about how to debug climate models at BSC. The part concerning the post-processing of experiment also need to be completed.

First step: Use autosubmit

To launch an experiment at BSC, for example a simulation with the ocean-atmosphere coupled version of EC-Earth, we have at our disposal autosubmit, a tools designed to run any models, and in particular climate models, on any machines. Use the documentation of autosubmit to get an experiment ID, create an experiment with the model version that you need, and to launch your experiment: http://autosubmit.readthedocs.io/en/latest/index.html. Autosubmit will provide you a name for your new experiment.

NB1: the information needed by autosubmit to prepare an experiment (model version, HPC, etc…) is set up in the file expdef_${exp}.conf. In particular, you have to indicate which model sources you want to use. At BSC, each model version appears generally under a git project: for example: https://earth.bsc.es/gitlab/es/auto-ecearth3 for the 2016 version.

NB2: to launch an experiment from BSC, do not do it from your local machine, but from the machine bscesautosubmit01 (to avoid overloading your poor own machine). ⇒ type ssh -XY bscesautosubmit01 to open the connection to this machine.

Second step: configuration

Then, you have to configure your experiment. After the creation of your experiment, you will find a new directory corresponding to your experiment in /esnas/autosubmit/ including:

* pkl/ a directory used by autosubmit to “store” the work flow of your experiments.

* plot: a directory where autosubmit generates plots to monitor your experiment.

* tmp: a temporary directory where autosubmit will run the local setup of your experiment. You will have a look in this directory if the Local Setup of your experiment fails.

* conf/: contains configurations files that you will need to fill before running your experiment:

  • autosubmit_${exp}.conf: autosubmit information. Generally, you will not modify this file that define in particular the version of autosubmit that you are using.
  • jobs_${exp}.conf: the information required by autosubmit when it will submit your job (requested time, total number of processor). For EC-Earth version anterior to EC-Earth 3.2Beta, you can use this table to find the number of processors and wall clock you have to put for each jobs: https://earth.bsc.es/wiki/doku.php?id=scaling:coupled_ec_earth . For EC_earth 3.2Beta: https://earth.bsc.es/wiki/doku.php?id=scaling:coupled_ec_earth_3.2_beta .
  • platforms_${exp}.conf: information related to the way to consume the computing time for your simulation. On which project can you afford such a simulation? You can specify a queue test if you want to test the new development that you are implementing.
  • proj_${exp}.conf : resolution, number of processors for each component of the model, initial conditions, options of the different components (atmospheric, oceanic models and coupling interface).

* proj: contains the directory model/ with all the source files of your climate model. This directory also contains the templates files. There is a version of templates for each HPC and each model version, since these files are used to transmit the information from the configuration files (that are in the conf/ directory) to the code during a simulation. You will find the different namelists defining the values of the flags or variables used by the different components of EC-Earth. For example, the namelists needed by the physical parametrisation in IFS are in the following directory:
/esnas/autosubmit/[exp]/git/model/sources/sources/ifs-36r4/src/ifs/phys_ec/sucldp.F90

Small configuration issues - May 2016

* BSC network is too slow for the current autosubmit current version. To avoid untimely failures, you have to specify in your .bashrc:
export SAGA_PTY_SSH_TIMEOUT=90
* When running on ecmwf or marenostrum model version older than EC-Eearth3.1f, you have to create a directory with the name of your experiment on your scratch /gpfs/scratch/bsc32/bsc[your_account] before launching your experiment or alternatively modify the localsetup.sh plugins by replacing putting a “/” at the end of DEST=$SCRATCH_DIR/$HPCPROJ/$HPCUSER/$EXPID .

Third step post-processing

To be completed.

Eventual step: debugging

When the model fails, it is recommended to have a look at the outputs files of EC-Earth, and in particular:

NEMO output: ocean.output

You will find a funny “AAAAAAAA” when the simulation didn't fail at the end of this output file. Otherwise, you have to look for “E R R O R” somewhere in this file to understand what's happened (do not forget the spaces between the letters of E R R O R).

If you want to check the last nemo file before a crash, you can assemble the results of the different processors in one single file (in the HPC directory where run the code). For example, on marenostrum3 with nemo running on 16 processors:

#########################
cp /gpfs/projects/bsc32/repository/apps/rebuild_nemo/rebuild_nemo* &
/gpfs/scratch/bsc32/bsc[your_account]/[exp]/[date]/[memb]/.
ldd rebuild_nemo.exe
./rebuild_nemo output.abort 16
ncview output.abort.nc
#########################

IFS output: NODE.* If the simulation is correctly finished, you will find at the end of this output file “End of Heap Utilization Profile”. Otherwise, look for “error” in the file.

tools/run_1st_exp.1464704119.txt.gz · Last modified: 2016/05/31 14:15 by cprodhom