How to trace using Extrae experiments on Autosubmit

1. Once you have created (or cloned) your project, go to:

cd experiment_root_path/proj/auto-ecearth3/sources/runtime/autosubmit

Modify ece-ifs+nemo.sh ( or ece-nemo.sh for Nemo standalone ) script to add the Extrae trace.sh call before executing the models

vi ece-ifs+nemo.sh +422

  # --------------------------------------
  # *** Start the run
  # --------------------------------------
  # Use the launch function from the platform configuration file
  t1=$(date +%s)
  launch \
${nem_numproc} /path/to/trace.sh ${nem_exe_file} -- \
      ${ifs_numproc} /path/to/trace.sh ${ifs_exe_file} -v ecmwf -e ${exp_name} -- \
      ${rnf_numproc} /path/to/trace.sh ${rnf_exe_file} -- \
      ${xio_numproc} /path/to/trace.sh ${xio_exe_file}
  ...

After that we must modify the platform configuration file where the launch function was declared.

cd experiment_root_path/proj/auto-ecearth3/sources/runtime/autosubmit/platform

Modify launch function on marenostrum4.cfg.tmpl to look like:

vi marenostrum4.cfg.tmpl +64

function launch() {

  cmd="mpirun"
  while (( "$#" ))
  do
      # Get number of MPI ranks and executable name
      nranks=$1
      trace=$2
      executable=./$(basename $3)
      shift
      shift
      shift
      cmd+=" -np $nranks $trace $executable"
      ...

2. In order to use Extrae package, you need to add it before the execution. One way you could do this is modifying the trace.sh script that you want to use adding the needed modules for the remote platform.

vi path/to/used/trace.sh

#!/bin/bash
 module add gcc/7.2.0
 module add EXTRAE/3.5.4
 source /apps/BSCTOOLS/extrae/3.5.4/impi_2017_4/etc/extrae.sh
 export EXTRAE_CONFIG_FILE=/gpfs/scratch/bsc32/bsc32442/extrae.xml
 export LD_PRELOAD=${EXTRAE_HOME}/lib/libmpitracecf.so # For C/F90 apps
 ## Run the desired program
 $*

3. Run your experiment as usual. Check that all changes are done in the remote directory before the SIM job start.