Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • autosubmit autosubmit
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 338
    • Issues 338
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 21
    • Merge requests 21
  • Deployments
    • Deployments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • Earth SciencesEarth Sciences
  • autosubmitautosubmit
  • Merge requests
  • !407

Calendar splits - rebased

  • Review changes

  • Download
  • Patches
  • Plain diff
Merged dbeltran requested to merge Destine-1177-rebased into master Mar 05, 2024
  • Overview 7
  • Commits 41
  • Pipelines 0
  • Changes 23

Added a calendar for splits #1177 (closed).

Added the following parameters:

JOBS:
 JOB:
    SPLITSIZE: 6 # default 1, takes priority
    SPLITSIZEUNIT: # default CHUNKSIZEUNIT - 1 {Year-1 == month, month-1 == day, day-1 == hour}, takes priority
EXPERIMENT:
 SPLITSIZE: # Default 1
 SPLITSIZEUNIT: # Default CHUNKSIZEUNIT -1

The job needs to be running as DATE, at least. When working under CHUNK it will take the startdate from CHUNK.

Example:

EXPERIMENT:
  DATELIST: '20000101'
  CHUNKSIZEUNIT: day
  CHUNKSIZE: '4'
  NUMCHUNKS: '2'
  CALENDAR: standard
JOBS:
  A:
   FILE: a.sh
   RUNNING: chunk
   SPLITS: 12
   SPLITSIZE: "2"

Test expected Results:

            next_start = "00"
            for i,job in enumerate(job_list[0:12]): # CHUNK 1
                parameters = job.update_parameters(config, parameters)
                end_hour = str(parameters['SPLIT'] * splitsize ).zfill(2)
                if end_hour == "24":
                    end_hour = "00"
                self.assertEqual(parameters['SPLIT'], i+1)
                self.assertEqual(parameters['SPLITSIZE'], splitsize)
                self.assertEqual(parameters['SPLITSIZEUNIT'], 'hour')
                self.assertEqual(parameters['SPLIT_START_DATE'], '20000101')
                self.assertEqual(parameters['SPLIT_START_YEAR'], '2000')
                self.assertEqual(parameters['SPLIT_START_MONTH'], '01')
                self.assertEqual(parameters['SPLIT_START_DAY'], '01')
                self.assertEqual(parameters['SPLIT_START_HOUR'], next_start)
                if parameters['SPLIT'] == 12:
                    self.assertEqual(parameters['SPLIT_END_DATE'], '20000102')
                    self.assertEqual(parameters['SPLIT_END_DAY'], '02')
                    self.assertEqual(parameters['SPLIT_END_DATE'], '20000102')
                    self.assertEqual(parameters['SPLIT_END_DAY'], '02')
                    self.assertEqual(parameters['SPLIT_END_YEAR'], '2000')
                    self.assertEqual(parameters['SPLIT_END_MONTH'], '01')
                    self.assertEqual(parameters['SPLIT_END_HOUR'], end_hour)
                else:
                    self.assertEqual(parameters['SPLIT_END_DATE'], '20000101')
                    self.assertEqual(parameters['SPLIT_END_DAY'], '01')
                    self.assertEqual(parameters['SPLIT_END_YEAR'], '2000')
                    self.assertEqual(parameters['SPLIT_END_MONTH'], '01')
                    self.assertEqual(parameters['SPLIT_END_HOUR'], end_hour)
                next_start = parameters['SPLIT_END_HOUR']

            next_start = "00"
            for i,job in enumerate(job_list[12:24]): # CHUNK 2
                parameters = job.update_parameters(config, parameters)
                end_hour = str(parameters['SPLIT'] * splitsize ).zfill(2)
                if end_hour == "24":
                    end_hour = "00"
                self.assertEqual(parameters['SPLIT'], i+1)
                self.assertEqual(parameters['SPLITSIZE'], splitsize)
                self.assertEqual(parameters['SPLITSIZEUNIT'], 'hour')
                self.assertEqual(parameters['SPLIT_START_DATE'], '20000105')
                self.assertEqual(parameters['SPLIT_START_YEAR'], '2000')
                self.assertEqual(parameters['SPLIT_START_MONTH'], '01')
                self.assertEqual(parameters['SPLIT_START_DAY'], '05')
                self.assertEqual(parameters['SPLIT_START_HOUR'], next_start)
                if parameters['SPLIT'] == 12:
                    self.assertEqual(parameters['SPLIT_END_DATE'], '20000106')
                    self.assertEqual(parameters['SPLIT_END_DAY'], '06')
                    self.assertEqual(parameters['SPLIT_END_YEAR'], '2000')
                    self.assertEqual(parameters['SPLIT_END_MONTH'], '01')
                    self.assertEqual(parameters['SPLIT_END_HOUR'], end_hour)
                else:
                    self.assertEqual(parameters['SPLIT_END_DATE'], '20000105')
                    self.assertEqual(parameters['SPLIT_END_DAY'], '05')
                    self.assertEqual(parameters['SPLIT_END_YEAR'], '2000')
                    self.assertEqual(parameters['SPLIT_END_MONTH'], '01')
                    self.assertEqual(parameters['SPLIT_END_HOUR'], end_hour)
                next_start = parameters['SPLIT_END_HOUR']

Added "auto" keyword.

JOBS:
  JOB:
    SPLITS: auto

When used, the splits generated will be automatic based on the calendar.

IF the possible_splits/splitsize is a float, one extra split will be generated for that date_chunk.

Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: Destine-1177-rebased