[enhancement] Allow recovery to update current running/ready jobs
From Slack thread with @dbeltran and @mcastril , and from issue in DestinE from @kkeller.
From @dbeltran:
So, we're missing an option in the recovery to update current waiting/ready jobs and ignore the other status.
We could modify Autosubmit to allow recovery
to be used when you have jobs running, without cancelling them. From @mcastril's comments:
- We have to consider how to map active jobs and workflow tasks, e.g. when job ids do not correspond to the ones in pkl, who can we know which ones are submitted, running...
- We have to find a solution that does not involve submitting jobs if we already have one running
- Maybe we could look at the scheduler output if there are active jobs, and see if all match the job ids in the DB, and update their statuses. If there's a mismatch, then users would have to force
-f
We also need to verify if there was no other reason why this feature was coded like this.
(Tried to summarize the comments here, feel free to edit if I added anything wrongly