- Added splits_to and splits_from
- Fixed some bugs with the function that goes through all scripts settings
- Refactored a little
EXAMPLES:
Complete Syntax:
JOBS:
DATA_NOTIFY:
...
RUNNING: DATE
SPLITS: 2
OPA:
DEPENDENCIES:
DATA_NOTIFY:
DATES_FROM:
ALL:
MEMBERS_FROM:
ALL:
CHUNKS_FROM:
1:
SPLITS_FROM:
1:
SPLITS_TO: "1"
2:
SPLITS_TO: "2"
2:
SPLITS_FROM:
1:
SPLITS_TO: "1"
2:
SPLITS_TO: "2"Simplified syntax example:
Differential:
FILE: CLEAN.sh
DEPENDENCIES:
DATA_NOTIFY:
CHUNKS_FROM:
ALL:
SPLITS_FROM:
1:
SPLITS_TO: "1"
2:
SPLITS_TO: "2"It is equivalent to:
Differential:
FILE: CLEAN.sh
DEPENDENCIES:
DATA_NOTIFY:
SPLITS_FROM:
1:
SPLITS_TO: "1"
2:
SPLITS_TO: "2"A reminder of the Algorithm logic:
First, it parses the "FROM" filters:
I'm the section "XY" I'll look into the "FROM_FILTERS" to see which tasks I must touch. It will follow a "natural" logic if a task is not there. ( per example, SIM_chunk_1 -> POST_chunk_1 )
The XY section, will check in order, it's a date string, then it's member string, then it's a chunk number, and finally, it split number. The order is always the same, but the filters can be skipped; per example, DATE -> MEMBER is the standard logic, but DATE -> SPLIT is also allowed. What is not permitted is to look in the reverse order, for example, SPLIT -> DATE.
Second, once the section knows exactly which task it must touch, it just will look to the FILTERS_TO subsections of the current level. If some subsection is not specified, it will look to the last level until no levels are left. ( default: natural )
Here it will have listed already the tasks it must touch. For ex a section running = DATE and SPLITS: "2"
{
XY { date: 2000, member: None, chunks: NONE, splits: 1} -> natural
XY { date: 2000, member: None, chunks: none, splits: 2} -> natural
XY { date: 2001, member: None, chunks: none, splits: 1} -> natural
XY { date: 2001, member: None, chunks: none, splits: 2} -> natural
}With "_TO " filters, we change this "natural" relation and pinpoint the tasks we want.
AB:
RUNNING: CHUNK
SPLITS: 2
XY:
RUNNING: ONCE
SPLITS: 2
DEPENDENCIES:
AB:
SPLITS_FROM:
1:
date_to: "2000"
members_to: "fc1"
chunks_to: "1,2"
splits_to: "2"
{
XY { date: None, member: None, chunks: NONE, splits: 1} -> AB { date: 2000, member: fc1, chunks: [1,2], splits: 2}
XY { date: None, member: None, chunks: none, splits: 2} -> natural
XY { date: None, member: None, chunks: none, splits: 1} -> AB { date: 2000, member: fc1, chunks: [1,2], splits: 2}
XY { date: None, member: None, chunks: none, splits: 2} -> natural
}