allow to specify TOTALJOBS for each platform
In expid a3ds @iayan is running remote jobs using the ifs_initial_conditions package on ecmwf-ecgate (retrieval of data from mars) and ecmwf-cca (for interpolation of ICs).
Be cause of restrictions on the ecgate SCRATCH as well as concurrent mars requests, we cannot run more than 3 concurrent retrieval job. The only way to achieve this is to set the global TOTALJOBS variable to 3.
However, this slows down the workflow because the interpolation jobs on cca take much longer than the retrieval, and are also limited to 3 concurrent jobs.
The workaround employed by @iayan is to process a number of the retrieval jobs, once those are done she changes the TOTALJOBS to a higher value of 20 and runs the interpolation jobs. This requires lots of user intervention in a task which could in theory be done without any intervention, if we could specify TOTALJOBS for each platform.
See details in https://earth.bsc.es/gitlab/es/ifs_initial_conditions/-/issues/12
FYI @mcastril