Operational workflows: Add recipe maker and crontab wrapper script to launch jobs automatically
Hi @nperez,
I have created a branch named dev-test_operational
to do some tests regarding creating and running recipes operationally. Right now it contains two new scripts:
-
recipe_maker.R
: It reads an existing recipe template provided by the user that contains all the details of the workflow except for the start date, which is an input parameter. It modifies the start date of the template and then creates a new recipe with the suffix_oper_YYYYMMDD
. This script is meant to be called from crontab-wrapper.sh but it can also be called on it own (seeRscript recipe_maker.R --help
for details) -
crontab-wrapper.sh
: It is a wrapper for recipe_maker.R + launch_SUNSET.sh + Autosubmit create, refresh, and run commands. It can be called directly from a CRON. Everything is done automatically so user needs to have passwordless ssh set up for the autosubmit01 machine. The usage for now is:
bash crontab-wrapper.sh --expid=a6wq --recipe=recipe_oper_test.yml --horizon=seasonal --sd=20241001
Where -sd is the start date; it can be left empty, in which case it is the first day of the current month (for seasonal) or the current day (for subseasonal). So for example, a CRON could be set up like:
00 10 12 * * bash /esarchive/scratch/vagudets/oper/sunset/crontab-wrapper.sh --expid=a6wq --recipe=recipe_oper_test.yml --horizon=seasonal
to run on the 12th of each month at 10 am.
Some details need to be refined and we can make it more sophisticated if need be but I think this should be a good start. Let me know what you think and if you are interested in trying it out.
Cheers,
Victòria