Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • SUNSET SUNSET
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 43
    • Issues 43
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 12
    • Merge requests 12
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Terraform modules
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Earth SciencesEarth Sciences
  • SUNSETSUNSET
  • Issues
  • #155
Closed
Open
Issue created Oct 25, 2024 by vagudets@vagudetsMaintainer

Operational workflows: Add recipe maker and crontab wrapper script to launch jobs automatically

Hi @nperez,

I have created a branch named dev-test_operational to do some tests regarding creating and running recipes operationally. Right now it contains two new scripts:

  1. recipe_maker.R: It reads an existing recipe template provided by the user that contains all the details of the workflow except for the start date, which is an input parameter. It modifies the start date of the template and then creates a new recipe with the suffix _oper_YYYYMMDD. This script is meant to be called from crontab-wrapper.sh but it can also be called on it own (see Rscript recipe_maker.R --help for details)
  2. crontab-wrapper.sh: It is a wrapper for recipe_maker.R + launch_SUNSET.sh + Autosubmit create, refresh, and run commands. It can be called directly from a CRON. Everything is done automatically so user needs to have passwordless ssh set up for the autosubmit01 machine. The usage for now is:

bash crontab-wrapper.sh --expid=a6wq --recipe=recipe_oper_test.yml --horizon=seasonal --sd=20241001

Where -sd is the start date; it can be left empty, in which case it is the first day of the current month (for seasonal) or the current day (for subseasonal). So for example, a CRON could be set up like:

00 10 12 * * bash /esarchive/scratch/vagudets/oper/sunset/crontab-wrapper.sh --expid=a6wq --recipe=recipe_oper_test.yml --horizon=seasonal

to run on the 12th of each month at 10 am.

Some details need to be refined and we can make it more sophisticated if need be but I think this should be a good start. Let me know what you think and if you are interested in trying it out.

Cheers,

Victòria

Edited Oct 25, 2024 by vagudets
Assignee
Assign to
Time tracking