How to handle data downloads every 5 minutes
As I think that Francesca explained to you today, for auto-PHENOMENA we will have to download data from a webpage every 5 minutes. We were thinking on how to set that up with Autosubmit.
My first idea would be to use RETRY_DELAY_TIME + soft dependency (FAILED). That would mean:
- We start the experiment, let's say, at 00:00. Data is downloaded. Job successful -> Next chunk submitted.
- The next chunk is submitted at 00:00+run time of the previous chunk. If more than 5 minutes passed, the task is failed (the data was already downloaded). Autosubmit waits for something like 3 minutes (RETRY_DELAY_TIME) to submit the retrial. In the meantime, it submits the next chunk, that will fail, wait for 3 minutes, and try again.
I see a lot of gaps in my approach. For example, if the first chunk takes more than 5 minutes to run, we would be losing data. But if the soft dependency is with RUNNING, we will have too many "attempts" at the same time, I think.
I believe that @bdepaula had some ideas!
cc @lherrero @fmacchia @ctena @cpinero @avinas (am I missing someone?)