Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • autosubmit autosubmit
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 338
    • Issues 338
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 21
    • Merge requests 21
  • Deployments
    • Deployments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • Earth SciencesEarth Sciences
  • autosubmitautosubmit
  • Merge requests
  • !339

Destine 1023

  • Review changes

  • Download
  • Patches
  • Plain diff
Merged dbeltran requested to merge DestinE-1023 into master May 15, 2023
  • Overview 42
  • Commits 30
  • Pipelines 0
  • Changes 5
  • Added splits_to and splits_from
  • Fixed some bugs with the function that goes through all scripts settings
  • Refactored a little

EXAMPLES:

Complete Syntax:

JOBS:
  DATA_NOTIFY:
    ...
    RUNNING: DATE
    SPLITS: 2
  OPA:
    DEPENDENCIES:
      DATA_NOTIFY:
        DATES_FROM:
          ALL:
            MEMBERS_FROM:
              ALL:
                CHUNKS_FROM:
                  1:
                    SPLITS_FROM:
                      1:
                        SPLITS_TO: "1"
                      2:
                        SPLITS_TO: "2"
                  2:
                    SPLITS_FROM:
                      1:
                        SPLITS_TO: "1"
                      2:
                        SPLITS_TO: "2"

Simplified syntax example:

  Differential:
    FILE: CLEAN.sh
    DEPENDENCIES:
      DATA_NOTIFY:
       CHUNKS_FROM: 
        ALL:
         SPLITS_FROM:
         1:
          SPLITS_TO: "1"
         2:
          SPLITS_TO: "2"

It is equivalent to:

  Differential:
    FILE: CLEAN.sh
    DEPENDENCIES:
      DATA_NOTIFY:
       SPLITS_FROM:
        1:
         SPLITS_TO: "1"
        2:
         SPLITS_TO: "2"

A reminder of the Algorithm logic:

First, it parses the "FROM" filters:

I'm the section "XY" I'll look into the "FROM_FILTERS" to see which tasks I must touch. It will follow a "natural" logic if a task is not there. ( per example, SIM_chunk_1 -> POST_chunk_1 )

The XY section, will check in order, it's a date string, then it's member string, then it's a chunk number, and finally, it split number. The order is always the same, but the filters can be skipped; per example, DATE -> MEMBER is the standard logic, but DATE -> SPLIT is also allowed. What is not permitted is to look in the reverse order, for example, SPLIT -> DATE.

Second, once the section knows exactly which task it must touch, it just will look to the FILTERS_TO subsections of the current level. If some subsection is not specified, it will look to the last level until no levels are left. ( default: natural )

Here it will have listed already the tasks it must touch. For ex a section running = DATE and SPLITS: "2"

 {
  XY { date: 2000, member: None, chunks: NONE, splits: 1} -> natural
  XY { date: 2000, member: None, chunks: none, splits: 2} -> natural
  XY { date: 2001, member: None, chunks: none, splits: 1} -> natural
  XY { date: 2001, member: None, chunks: none, splits: 2} -> natural
 }

With "_TO " filters, we change this "natural" relation and pinpoint the tasks we want.

AB:
 RUNNING: CHUNK
 SPLITS: 2
XY:
 RUNNING: ONCE
 SPLITS: 2
 DEPENDENCIES:
  AB:
   SPLITS_FROM:
    1:
     date_to: "2000"
     members_to: "fc1"
     chunks_to: "1,2"
     splits_to: "2"
 {
  XY { date: None, member: None, chunks: NONE, splits: 1} -> AB { date: 2000, member: fc1, chunks: [1,2], splits: 2}
  XY { date: None, member: None, chunks: none, splits: 2} -> natural
  XY { date: None, member: None, chunks: none, splits: 1} -> AB { date: 2000, member: fc1, chunks: [1,2], splits: 2}
  XY { date: None, member: None, chunks: none, splits: 2} -> natural
 }
Edited May 15, 2023 by dbeltran
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: DestinE-1023