Use Autosubmit as workflow manager
Hi @bdepaula
I open this issue to track the development of using startR with autosubmit.
I ran a basic startR script, which includes a data array with 7 dimensions, and it is chunked into 8 chunks, along 3 dimensions "latitude", "longitude", "sdate".
attributes(data)$Dimensions
dat var sdate ensemble time latitude longitude
1 1 2 51 7 640 1296
str(chunks)
List of 3
$ latitude : num 2
$ longitude: num 2
$ sdate : num 2
In ecFlow UI, the chunking is like the below (I didn't open all the layers, but the rest has the same structure as the first "Chunk".)
You can see all the ecFlow-related scripts here: https://earth.bsc.es/gitlab/aho/aho-testtest/-/tree/master/startR/issue/issue177/script_ecFlow or under this path /home/Earth/aho/startR_local/STARTR_CHUNKING_4737920362/
Some notes:
-
ByChunks() creates
load_process_save_chunk.R
(the full startR script),Chunk.ecf
(pastinghead.h
,slurm.h
,tail.h
,load_process_save_chunk.R
together), andstartR_chunking.def
(define the family).dat_CHUNK_1/
is the folder of first chunk, which includesChunk.job1
, theChunk.ecf
for the first chunk. - To make it less complicated, we can create another ByChunks() for autosubmit, instead of directly including the development in ByChunks().
Please let me know if anything is not clear. Thanks for your help!
Best,
An-Chi