Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • autosubmit autosubmit
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 338
    • Issues 338
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 21
    • Merge requests 21
  • Deployments
    • Deployments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • Earth SciencesEarth Sciences
  • autosubmitautosubmit
  • Issues
  • #645
Closed
Open
Issue created Jan 21, 2021 by Etienne Tourigny@etourignDeveloper

allow to specify TOTALJOBS for each platform

In expid a3ds @iayan is running remote jobs using the ifs_initial_conditions package on ecmwf-ecgate (retrieval of data from mars) and ecmwf-cca (for interpolation of ICs).

Be cause of restrictions on the ecgate SCRATCH as well as concurrent mars requests, we cannot run more than 3 concurrent retrieval job. The only way to achieve this is to set the global TOTALJOBS variable to 3.

However, this slows down the workflow because the interpolation jobs on cca take much longer than the retrieval, and are also limited to 3 concurrent jobs.

The workaround employed by @iayan is to process a number of the retrieval jobs, once those are done she changes the TOTALJOBS to a higher value of 20 and runs the interpolation jobs. This requires lots of user intervention in a task which could in theory be done without any intervention, if we could specify TOTALJOBS for each platform.

See details in https://earth.bsc.es/gitlab/es/ifs_initial_conditions/-/issues/12

FYI @mcastril

Assignee
Assign to
Time tracking