Added a calendar for splits #1177.
Added the following parameters:
JOBS:
JOB:
SPLITSIZE: 6 # default 1, takes priority
SPLITSIZEUNIT: # default CHUNKSIZEUNIT - 1 {Year-1 == month, month-1 == day, day-1 == hour}, takes priority
EXPERIMENT:
SPLITSIZE: # Default 1
SPLITSIZEUNIT: # Default CHUNKSIZEUNIT -1
The job needs to be running as DATE, at least. When working under CHUNK it will take the startdate from CHUNK.
Example:
EXPERIMENT:
DATELIST: '20000101'
CHUNKSIZEUNIT: day
CHUNKSIZE: '4'
NUMCHUNKS: '2'
CALENDAR: standard
JOBS:
A:
FILE: a.sh
RUNNING: chunk
SPLITS: 12
SPLITSIZE: "2"
Test expected Results:
next_start = "00"
for i,job in enumerate(job_list[0:12]): # CHUNK 1
parameters = job.update_parameters(config, parameters)
end_hour = str(parameters['SPLIT'] * splitsize ).zfill(2)
if end_hour == "24":
end_hour = "00"
self.assertEqual(parameters['SPLIT'], i+1)
self.assertEqual(parameters['SPLITSIZE'], splitsize)
self.assertEqual(parameters['SPLITSIZEUNIT'], 'hour')
self.assertEqual(parameters['SPLIT_START_DATE'], '20000101')
self.assertEqual(parameters['SPLIT_START_YEAR'], '2000')
self.assertEqual(parameters['SPLIT_START_MONTH'], '01')
self.assertEqual(parameters['SPLIT_START_DAY'], '01')
self.assertEqual(parameters['SPLIT_START_HOUR'], next_start)
if parameters['SPLIT'] == 12:
self.assertEqual(parameters['SPLIT_END_DATE'], '20000102')
self.assertEqual(parameters['SPLIT_END_DAY'], '02')
self.assertEqual(parameters['SPLIT_END_DATE'], '20000102')
self.assertEqual(parameters['SPLIT_END_DAY'], '02')
self.assertEqual(parameters['SPLIT_END_YEAR'], '2000')
self.assertEqual(parameters['SPLIT_END_MONTH'], '01')
self.assertEqual(parameters['SPLIT_END_HOUR'], end_hour)
else:
self.assertEqual(parameters['SPLIT_END_DATE'], '20000101')
self.assertEqual(parameters['SPLIT_END_DAY'], '01')
self.assertEqual(parameters['SPLIT_END_YEAR'], '2000')
self.assertEqual(parameters['SPLIT_END_MONTH'], '01')
self.assertEqual(parameters['SPLIT_END_HOUR'], end_hour)
next_start = parameters['SPLIT_END_HOUR']
next_start = "00"
for i,job in enumerate(job_list[12:24]): # CHUNK 2
parameters = job.update_parameters(config, parameters)
end_hour = str(parameters['SPLIT'] * splitsize ).zfill(2)
if end_hour == "24":
end_hour = "00"
self.assertEqual(parameters['SPLIT'], i+1)
self.assertEqual(parameters['SPLITSIZE'], splitsize)
self.assertEqual(parameters['SPLITSIZEUNIT'], 'hour')
self.assertEqual(parameters['SPLIT_START_DATE'], '20000105')
self.assertEqual(parameters['SPLIT_START_YEAR'], '2000')
self.assertEqual(parameters['SPLIT_START_MONTH'], '01')
self.assertEqual(parameters['SPLIT_START_DAY'], '05')
self.assertEqual(parameters['SPLIT_START_HOUR'], next_start)
if parameters['SPLIT'] == 12:
self.assertEqual(parameters['SPLIT_END_DATE'], '20000106')
self.assertEqual(parameters['SPLIT_END_DAY'], '06')
self.assertEqual(parameters['SPLIT_END_YEAR'], '2000')
self.assertEqual(parameters['SPLIT_END_MONTH'], '01')
self.assertEqual(parameters['SPLIT_END_HOUR'], end_hour)
else:
self.assertEqual(parameters['SPLIT_END_DATE'], '20000105')
self.assertEqual(parameters['SPLIT_END_DAY'], '05')
self.assertEqual(parameters['SPLIT_END_YEAR'], '2000')
self.assertEqual(parameters['SPLIT_END_MONTH'], '01')
self.assertEqual(parameters['SPLIT_END_HOUR'], end_hour)
next_start = parameters['SPLIT_END_HOUR']
Added "auto" keyword.
JOBS:
JOB:
SPLITS: auto
When used, the splits generated will be automatic based on the calendar.
IF the possible_splits/splitsize is a float, one extra split will be generated for that date_chunk.