Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • startR startR
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 29
    • Issues 29
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 7
    • Merge requests 7
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Earth SciencesEarth Sciences
  • startRstartR
  • Issues
  • #177
Closed
Open
Issue created Jun 02, 2023 by aho@ahoMaintainer

Use Autosubmit as workflow manager

Hi @bdepaula

I open this issue to track the development of using startR with autosubmit.

I ran a basic startR script, which includes a data array with 7 dimensions, and it is chunked into 8 chunks, along 3 dimensions "latitude", "longitude", "sdate".

attributes(data)$Dimensions
      dat       var     sdate  ensemble      time  latitude longitude 
        1         1         2        51         7       640      1296 

str(chunks)
List of 3
 $ latitude : num 2
 $ longitude: num 2
 $ sdate    : num 2

In ecFlow UI, the chunking is like the below (I didn't open all the layers, but the rest has the same structure as the first "Chunk".) image

You can see all the ecFlow-related scripts here: https://earth.bsc.es/gitlab/aho/aho-testtest/-/tree/master/startR/issue/issue177/script_ecFlow or under this path /home/Earth/aho/startR_local/STARTR_CHUNKING_4737920362/

Some notes:

  • ByChunks() creates load_process_save_chunk.R (the full startR script), Chunk.ecf (pasting head.h, slurm.h, tail.h, load_process_save_chunk.R together), and startR_chunking.def (define the family). dat_CHUNK_1/ is the folder of first chunk, which includes Chunk.job1, the Chunk.ecf for the first chunk.
  • To make it less complicated, we can create another ByChunks() for autosubmit, instead of directly including the development in ByChunks().

Please let me know if anything is not clear. Thanks for your help!

@vagudets @nperez

Best,
An-Chi

Assignee
Assign to
Time tracking