This is an old revision of the document!
How to perturb ocean restarts
When a problem of numerical instability occurs causing a crash of the model perturbation of the restarts can be useful. Here are presented the steps to follow to perturb ocean restarts. The script generates an arbitrary number of NEMO oceanic restarts that are copies of a reference, plus a perturbation.
How to perturb ocean restart: This script has been tested on MN3 Nord3 and Ecmwf
1- In order to perturb the restarts you need the script: gener_pertubation.bash, available in GitLab under CPG tool project (https://earth.bsc.es/gitlab/es/cpg_tools/tree/PerturbRestart/prep_restarts)
2-As practical example: consider that your chunk n3 failed for an instability problem, you have to perturb the restarts of the chunk n2. You can decide how many restarts you want to perturb, just one, of all of them. Normally you have many restarts as the number of processors you are using in NEMO.
So, go to the your run directory, make a backup of the restarts folder you want to perturb, just in case (with EC-Earth 3.2.2 the path is something like EXP_ID_/STARTDATE/fc#/runtime/NEMO_restart_#. Please note that with different version of EC-Earth, the path can be slightly different) and run gener_pertubation.bash specifying the name of the restarts and the n of restarts to perturb. For example:
gener_pertubation.bash a0aj_00019536_restart_oce_000.nc 3
It will automatically generate 3 perturbed restarts. Change the names of these new restarts to the original names and run your experiment from the last chuck that failed.
You may want to perturb all the restarts you have, you would like to do loop over all the restarts, perturb then and even change the names to the original ones.
In case you want to perturb all the restarts, here there is simple example of loop script that does the job:
#!/bin/bash # Loop on all the nemo ocean restart files nfiles=960 for num in `seq 0 $(( $nfiles -1 ))` ; do num3=$(printf "%03d" $num) file=a0bb_00078912_restart_oce_0${num3}.nc echo $file . gener_perturbation.bash $file 1 mv a0bb_00078912_restart_oce_0${num3}_fc0.nc $file done
In this session are collecting the error messages that can be found when a numerical instability is present. When these errors are present the perturbation of the restarts can solve the problem.
For further information ask any of the CP members or write to vsicardi@bsc.es