2021-12-16 13:15:19,015 
Checking configuration files...
[WARNING] 2021-12-16 13:15:19,021 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-16 13:15:19,022 expdef_a29z.conf OK
2021-12-16 13:15:19,022 platforms_a29z.conf OK
2021-12-16 13:15:19,023 jobs_a29z.conf OK
2021-12-16 13:15:19,024 wrappers OK
2021-12-16 13:15:19,024 autosubmit_a29z.conf OK
2021-12-16 13:15:19,024 Configuration files OK

2021-12-16 13:15:19,024 Autosubmit is running with 3.14.0b
2021-12-16 13:15:19,024 The a29z experiment 3.13.0 version is being updated to 3.14.0b for match autosubmit version
2021-12-16 13:15:19,030 Preparing .lock file to avoid multiple instances with same experiment id
2021-12-16 13:15:19,208 Custom directives from platform.conf: None
2021-12-16 13:15:19,208 Custom directives from platform.conf: None
2021-12-16 13:15:19,208 The Experiment name is: a29z
2021-12-16 13:15:19,208 Sleep: 10
2021-12-16 13:15:19,208 Default retrials: 0
2021-12-16 13:15:19,208 Starting job submission...
2021-12-16 13:15:19,217 Creating jobs...
2021-12-16 13:15:19,217 Loading JobList
2021-12-16 13:15:19,225 Creating LOCAL_SETUP jobs
2021-12-16 13:15:19,225 Creating REMOTE_SETUP jobs
2021-12-16 13:15:19,225 Creating INI jobs
2021-12-16 13:15:19,227 Creating SIM jobs
2021-12-16 13:15:19,241 Creating POST jobs
2021-12-16 13:15:19,255 Creating CLEAN jobs
2021-12-16 13:15:19,267 Creating TRANSFER jobs
2021-12-16 13:15:19,271 Creating LOCALPOST jobs
2021-12-16 13:15:19,272 Creating PLOT jobs
2021-12-16 13:15:19,273 Adding dependencies...
2021-12-16 13:15:19,273 Adding dependencies for LOCAL_SETUP jobs
2021-12-16 13:15:19,273 Adding dependencies for REMOTE_SETUP jobs
2021-12-16 13:15:19,273 Adding dependencies for INI jobs
2021-12-16 13:15:19,273 Adding dependencies for SIM jobs
2021-12-16 13:15:19,276 Adding dependencies for POST jobs
2021-12-16 13:15:19,277 Adding dependencies for CLEAN jobs
2021-12-16 13:15:19,279 Adding dependencies for TRANSFER jobs
2021-12-16 13:15:19,280 Adding dependencies for LOCALPOST jobs
2021-12-16 13:15:19,280 Adding dependencies for PLOT jobs
2021-12-16 13:15:19,280 Removing redundant dependencies...
2021-12-16 13:15:19,308 Using existing valid structure.
2021-12-16 13:15:19,309 Starting from job list restored from /esarchive/autosubmit/a29z/pkl files
2021-12-16 13:15:19,309 Length of the jobs list: 590
2021-12-16 13:15:19,309 Loading parameters...
2021-12-16 13:15:19,310 Parameters load.
2021-12-16 13:15:19,310 Checking experiment templates...
2021-12-16 13:15:19,310 Checking scripts...
2021-12-16 13:15:19,661 147 of 590 checked
2021-12-16 13:15:19,984 294 of 590 checked
2021-12-16 13:15:20,306 441 of 590 checked
2021-12-16 13:15:20,628 588 of 590 checked
2021-12-16 13:15:20,632 590 of 590 checked
2021-12-16 13:15:20,634 Scripts OK
2021-12-16 13:15:20,634 Loading job packages
2021-12-16 13:15:20,645 Checking job_list current status
2021-12-16 13:15:20,646 Updating FAILED jobs
2021-12-16 13:15:20,646 Updating WAITING jobs
2021-12-16 13:15:20,647 Update finished
2021-12-16 13:15:20,652 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-16 13:15:20,661 Job list saved
2021-12-16 13:15:20,665 Autosubmit is running with v3.14.0b
2021-12-16 13:15:20,666 Running job data structure
2021-12-16 13:15:21,007 Checking the connection to all platforms in use
2021-12-16 13:15:21,650 [transfer_node] Correct user privileges for host bscearth000
2021-12-16 13:15:23,990 [marenostrum4] Correct user privileges for host mn1.bsc.es
2021-12-16 13:15:23,990 Running main loop
2021-12-16 13:15:23,991 Reloading parameters...
2021-12-16 13:15:23,991 Loading parameters...
2021-12-16 13:15:23,992 Parameters load.
2021-12-16 13:15:23,992 

590 of 590 jobs remaining (13:15)
2021-12-16 13:15:23,992 Sleep: 10
2021-12-16 13:15:23,992 Number of retrials: 0
2021-12-16 13:15:23,992 WRAPPER CHECK TIME = 10
2021-12-16 13:15:23,996 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z_backup.pkl
2021-12-16 13:15:24,005 Job list saved
2021-12-16 13:15:24,011 Checking jobs for platform=transfer_node
2021-12-16 13:15:24,013 Checking jobs for platform=marenostrum4
2021-12-16 13:15:24,013 End of checking
2021-12-16 13:15:24,014 Updating FAILED jobs
2021-12-16 13:15:24,014 Updating WAITING jobs
2021-12-16 13:15:24,016 Update finished
2021-12-16 13:15:24,021 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-16 13:15:24,030 Job list saved
2021-12-16 13:15:24,038 
Jobs ready for transfer_node: 1
2021-12-16 13:15:24,040 Number of jobs available: 3
2021-12-16 13:15:24,040 Number of jobs ready: 1
2021-12-16 13:15:24,041 Jobs ready for transfer_node: 1
2021-12-16 13:15:24,044 Creating Scripts
2021-12-16 13:15:24,055 Sending Files
2021-12-16 13:15:24,323 /esarchive/scratch/Earth/wuruchi/a29z/LOG_a29z has been created on bscearth000 .
2021-12-16 13:15:24,540 Submitting
2021-12-16 13:15:24,545 a29z_LOCAL_SETUP_STAT have been removed
2021-12-16 13:15:24,548 a29z_LOCAL_SETUP_COMPLETED been removed
2021-12-16 13:15:24,768 Job ID: 7696

2021-12-16 13:15:24,768 a29z_LOCAL_SETUP submitted
2021-12-16 13:15:24,898 
Jobs ready for marenostrum4: 0
2021-12-16 13:15:24,900 Number of jobs available: 3
2021-12-16 13:15:24,900 Number of jobs ready: 0
2021-12-16 13:15:24,904 Updating FAILED jobs
2021-12-16 13:15:24,904 Updating WAITING jobs
2021-12-16 13:15:24,905 Update finished
2021-12-16 13:15:24,907 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-16 13:15:24,907 a29z_LOCAL_SETUP                   7696           SUBMITTED      transfer_node       serial         
2021-12-16 13:15:24,909 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-16 13:15:24,918 Job list saved
2021-12-16 13:15:34,934 Reloading parameters...
2021-12-16 13:15:34,934 Loading parameters...
2021-12-16 13:15:34,935 Parameters load.
2021-12-16 13:15:34,935 

590 of 590 jobs remaining (13:15)
2021-12-16 13:15:34,935 Sleep: 10
2021-12-16 13:15:34,935 Number of retrials: 0
2021-12-16 13:15:34,935 WRAPPER CHECK TIME = 10
2021-12-16 13:15:34,937 Checking jobs for platform=transfer_node
2021-12-16 13:15:37,158 Successful check job command: nohup kill -0 7696 > /dev/null 2>&1; echo $?
2021-12-16 13:15:37,158 a29z_LOCAL_SETUP job seems to have completed: checking...
2021-12-16 13:15:37,167 Job a29z_LOCAL_SETUP is COMPLETED
2021-12-16 13:15:37,178 a29z_LOCAL_SETUP_STAT file have been transfered
[WARNING] 2021-12-16 13:15:37,277 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-16 13:15:37,287 a29z_LOCAL_SETUP_STAT file have been transfered
2021-12-16 13:15:37,354 Checking jobs for platform=marenostrum4
2021-12-16 13:15:37,354 End of checking
2021-12-16 13:15:37,356 Updating FAILED jobs
2021-12-16 13:15:37,356 Updating WAITING jobs
2021-12-16 13:15:37,356 Setting job: a29z_REMOTE_SETUP status to: READY (all parents completed)...
2021-12-16 13:15:37,357 Update finished
2021-12-16 13:15:37,362 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-16 13:15:37,362 a29z_LOCAL_SETUP                   7696           COMPLETED      transfer_node       serial         
2021-12-16 13:15:37,364 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-16 13:15:37,373 Job list saved
2021-12-16 13:15:37,377 
Jobs ready for transfer_node: 1
2021-12-16 13:15:37,379 Number of jobs available: 3
2021-12-16 13:15:37,379 Number of jobs ready: 1
2021-12-16 13:15:37,380 Jobs ready for transfer_node: 1
2021-12-16 13:15:37,383 Creating Scripts
2021-12-16 13:15:37,394 Sending Files
2021-12-16 13:15:37,662 /esarchive/scratch/Earth/wuruchi/a29z/LOG_a29z has been created on bscearth000 .
2021-12-16 13:15:37,675 Submitting
2021-12-16 13:15:37,678 a29z_REMOTE_SETUP_STAT have been removed
2021-12-16 13:15:37,680 a29z_REMOTE_SETUP_COMPLETED been removed
2021-12-16 13:15:38,116 Job ID: 8214

2021-12-16 13:15:38,117 a29z_REMOTE_SETUP submitted
2021-12-16 13:15:38,250 
Jobs ready for marenostrum4: 0
2021-12-16 13:15:38,252 Number of jobs available: 3
2021-12-16 13:15:38,252 Number of jobs ready: 0
2021-12-16 13:15:38,255 Updating FAILED jobs
2021-12-16 13:15:38,256 Updating WAITING jobs
2021-12-16 13:15:38,257 Update finished
2021-12-16 13:15:38,261 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-16 13:15:38,261 a29z_LOCAL_SETUP                   7696           COMPLETED      transfer_node       serial         
2021-12-16 13:15:38,261 a29z_REMOTE_SETUP                  8214           SUBMITTED      transfer_node       serial         
2021-12-16 13:15:38,263 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-16 13:15:38,272 Job list saved
[WARNING] 2021-12-16 13:15:42,289 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-16 13:15:42,291 Custom directives from platform.conf: None
2021-12-16 13:15:42,292 Custom directives from platform.conf: None
2021-12-16 13:15:48,537 Reloading parameters...
2021-12-16 13:15:48,537 Loading parameters...
2021-12-16 13:15:48,538 Parameters load.
2021-12-16 13:15:48,538 

589 of 590 jobs remaining (13:15)
2021-12-16 13:15:48,538 Sleep: 10
2021-12-16 13:15:48,538 Number of retrials: 0
2021-12-16 13:15:48,538 WRAPPER CHECK TIME = 10
2021-12-16 13:15:48,540 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z_backup.pkl
2021-12-16 13:15:48,549 Job list saved
2021-12-16 13:15:48,563 Checking jobs for platform=transfer_node
2021-12-16 13:15:50,783 Successful check job command: nohup kill -0 8214 > /dev/null 2>&1; echo $?
2021-12-16 13:15:50,783 a29z_REMOTE_SETUP job seems to have completed: checking...
2021-12-16 13:15:50,790 Job a29z_REMOTE_SETUP is COMPLETED
2021-12-16 13:15:50,801 a29z_REMOTE_SETUP_STAT file have been transfered
[WARNING] 2021-12-16 13:15:50,876 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-16 13:15:51,071 a29z_REMOTE_SETUP_STAT file have been transfered
2021-12-16 13:15:51,144 Checking jobs for platform=marenostrum4
2021-12-16 13:15:51,144 End of checking
2021-12-16 13:15:51,144 Updating FAILED jobs
2021-12-16 13:15:51,145 Updating WAITING jobs
2021-12-16 13:15:51,145 Setting job: a29z_19550501_fc0_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19550501_fc1_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19550501_fc2_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19550501_fc3_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19550501_fc4_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19550501_fc5_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19550501_fc6_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19551101_fc0_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19551101_fc1_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19551101_fc2_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19551101_fc3_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19551101_fc4_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19551101_fc5_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,145 Setting job: a29z_19551101_fc6_INI status to: READY (all parents completed)...
2021-12-16 13:15:51,147 Update finished
2021-12-16 13:15:51,151 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-16 13:15:51,151 a29z_LOCAL_SETUP                   7696           COMPLETED      transfer_node       serial         
2021-12-16 13:15:51,152 a29z_REMOTE_SETUP                  8214           COMPLETED      transfer_node       serial         
2021-12-16 13:15:51,154 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-16 13:15:51,163 Job list saved
2021-12-16 13:15:51,167 
Jobs ready for transfer_node: 14
2021-12-16 13:15:51,169 Number of jobs available: 3
2021-12-16 13:15:51,169 Number of jobs ready: 14
2021-12-16 13:15:51,170 Jobs ready for transfer_node: 14
2021-12-16 13:15:51,173 Creating Scripts
2021-12-16 13:15:51,187 Sending Files
2021-12-16 13:15:51,454 /esarchive/scratch/Earth/wuruchi/a29z/LOG_a29z has been created on bscearth000 .
2021-12-16 13:15:51,466 Submitting
2021-12-16 13:15:51,469 a29z_19550501_fc0_INI_STAT have been removed
2021-12-16 13:15:51,472 a29z_19550501_fc0_INI_COMPLETED been removed
2021-12-16 13:15:51,771 Job ID: 8560

2021-12-16 13:15:51,771 a29z_19550501_fc0_INI submitted
2021-12-16 13:15:51,904 Creating Scripts
2021-12-16 13:15:51,918 Sending Files
2021-12-16 13:15:52,185 /esarchive/scratch/Earth/wuruchi/a29z/LOG_a29z has been created on bscearth000 .
2021-12-16 13:15:52,198 Submitting
2021-12-16 13:15:52,201 a29z_19550501_fc1_INI_STAT have been removed
2021-12-16 13:15:52,204 a29z_19550501_fc1_INI_COMPLETED been removed
2021-12-16 13:15:52,421 Job ID: 8737

2021-12-16 13:15:52,421 a29z_19550501_fc1_INI submitted
2021-12-16 13:15:52,572 Creating Scripts
2021-12-16 13:15:52,585 Sending Files
2021-12-16 13:15:52,853 /esarchive/scratch/Earth/wuruchi/a29z/LOG_a29z has been created on bscearth000 .
2021-12-16 13:15:53,242 Submitting
2021-12-16 13:15:53,247 a29z_19550501_fc2_INI_STAT have been removed
2021-12-16 13:15:53,250 a29z_19550501_fc2_INI_COMPLETED been removed
2021-12-16 13:15:53,467 Job ID: 8920

2021-12-16 13:15:53,468 a29z_19550501_fc2_INI submitted
2021-12-16 13:15:53,600 
Jobs ready for marenostrum4: 0
2021-12-16 13:15:53,602 Number of jobs available: 3
2021-12-16 13:15:53,602 Number of jobs ready: 0
2021-12-16 13:15:53,606 Updating FAILED jobs
2021-12-16 13:15:53,606 Updating WAITING jobs
2021-12-16 13:15:53,607 Update finished
2021-12-16 13:15:53,611 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-16 13:15:53,611 a29z_LOCAL_SETUP                   7696           COMPLETED      transfer_node       serial         
2021-12-16 13:15:53,611 a29z_REMOTE_SETUP                  8214           COMPLETED      transfer_node       serial         
2021-12-16 13:15:53,611 a29z_19550501_fc0_INI              8560           SUBMITTED      transfer_node       serial         
2021-12-16 13:15:53,611 a29z_19550501_fc1_INI              8737           SUBMITTED      transfer_node       serial         
2021-12-16 13:15:53,611 a29z_19550501_fc2_INI              8920           SUBMITTED      transfer_node       serial         
2021-12-16 13:15:53,613 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-16 13:15:53,622 Job list saved
[WARNING] 2021-12-16 13:15:55,891 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-16 13:15:55,893 Custom directives from platform.conf: None
2021-12-16 13:15:55,893 Custom directives from platform.conf: None
2021-12-16 13:16:03,708 Reloading parameters...
2021-12-16 13:16:03,708 Loading parameters...
2021-12-16 13:16:03,709 Parameters load.
2021-12-16 13:16:03,709 

588 of 590 jobs remaining (13:16)
2021-12-16 13:16:03,709 Sleep: 10
2021-12-16 13:16:03,709 Number of retrials: 0
2021-12-16 13:16:03,709 WRAPPER CHECK TIME = 10
2021-12-16 13:16:03,711 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z_backup.pkl
2021-12-16 13:16:03,720 Job list saved
2021-12-16 13:16:03,726 Checking jobs for platform=transfer_node
2021-12-16 13:16:05,997 Successful check job command: nohup kill -0 8560 > /dev/null 2>&1; echo $?
2021-12-16 13:16:05,997 a29z_19550501_fc0_INI job seems to have completed: checking...
2021-12-16 13:16:06,182 Job a29z_19550501_fc0_INI is COMPLETED
2021-12-16 13:16:06,233 a29z_19550501_fc0_INI_STAT file have been transfered
[WARNING] 2021-12-16 13:16:06,301 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-16 13:16:06,311 a29z_19550501_fc0_INI_STAT file have been transfered
2021-12-16 13:16:08,594 Successful check job command: nohup kill -0 8737 > /dev/null 2>&1; echo $?
2021-12-16 13:16:08,594 a29z_19550501_fc1_INI job seems to have completed: checking...
2021-12-16 13:16:08,819 Job a29z_19550501_fc1_INI is COMPLETED
2021-12-16 13:16:08,999 a29z_19550501_fc1_INI_STAT file have been transfered
[WARNING] 2021-12-16 13:16:09,062 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-16 13:16:09,073 a29z_19550501_fc1_INI_STAT file have been transfered
[WARNING] 2021-12-16 13:16:11,315 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-16 13:16:11,317 Custom directives from platform.conf: None
2021-12-16 13:16:11,317 Custom directives from platform.conf: None
2021-12-16 13:16:11,353 Successful check job command: nohup kill -0 8920 > /dev/null 2>&1; echo $?
2021-12-16 13:16:11,353 a29z_19550501_fc2_INI job seems to have completed: checking...
2021-12-16 13:16:11,459 Job a29z_19550501_fc2_INI is COMPLETED
2021-12-16 13:16:11,479 a29z_19550501_fc2_INI_STAT file have been transfered
[WARNING] 2021-12-16 13:16:11,544 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-16 13:16:11,554 a29z_19550501_fc2_INI_STAT file have been transfered
2021-12-16 13:16:11,620 Checking jobs for platform=marenostrum4
2021-12-16 13:16:11,621 End of checking
2021-12-16 13:16:11,621 Updating FAILED jobs
2021-12-16 13:16:11,621 Updating WAITING jobs
2021-12-16 13:16:11,622 Setting job: a29z_19550501_fc0_1_SIM status to: READY (all parents completed)...
2021-12-16 13:16:11,622 Setting job: a29z_19550501_fc1_1_SIM status to: READY (all parents completed)...
2021-12-16 13:16:11,622 Setting job: a29z_19550501_fc2_1_SIM status to: READY (all parents completed)...
2021-12-16 13:16:11,623 Update finished
2021-12-16 13:16:11,628 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-16 13:16:11,628 a29z_LOCAL_SETUP                   7696           COMPLETED      transfer_node       serial         
2021-12-16 13:16:11,628 a29z_REMOTE_SETUP                  8214           COMPLETED      transfer_node       serial         
2021-12-16 13:16:11,628 a29z_19550501_fc0_INI              8560           COMPLETED      transfer_node       serial         
2021-12-16 13:16:11,628 a29z_19550501_fc1_INI              8737           COMPLETED      transfer_node       serial         
2021-12-16 13:16:11,628 a29z_19550501_fc2_INI              8920           COMPLETED      transfer_node       serial         
2021-12-16 13:16:11,630 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-16 13:16:11,639 Job list saved
2021-12-16 13:16:11,652 
Jobs ready for transfer_node: 11
2021-12-16 13:16:11,654 Number of jobs available: 3
2021-12-16 13:16:11,655 Number of jobs ready: 11
2021-12-16 13:16:11,656 Jobs ready for transfer_node: 11
2021-12-16 13:16:11,659 Creating Scripts
2021-12-16 13:16:11,671 Sending Files
2021-12-16 13:16:11,939 /esarchive/scratch/Earth/wuruchi/a29z/LOG_a29z has been created on bscearth000 .
2021-12-16 13:16:12,117 Submitting
2021-12-16 13:16:12,120 a29z_19550501_fc3_INI_STAT have been removed
2021-12-16 13:16:12,122 a29z_19550501_fc3_INI_COMPLETED been removed
2021-12-16 13:16:12,339 Job ID: 9774

2021-12-16 13:16:12,340 a29z_19550501_fc3_INI submitted
2021-12-16 13:16:12,471 Creating Scripts
2021-12-16 13:16:12,484 Sending Files
2021-12-16 13:16:12,752 /esarchive/scratch/Earth/wuruchi/a29z/LOG_a29z has been created on bscearth000 .
2021-12-16 13:16:12,764 Submitting
2021-12-16 13:16:12,768 a29z_19550501_fc4_INI_STAT have been removed
2021-12-16 13:16:12,771 a29z_19550501_fc4_INI_COMPLETED been removed
2021-12-16 13:16:13,640 Job ID: 9960

2021-12-16 13:16:13,640 a29z_19550501_fc4_INI submitted
2021-12-16 13:16:13,955 Creating Scripts
2021-12-16 13:16:13,967 Sending Files
[WARNING] 2021-12-16 13:16:14,074 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-16 13:16:14,076 Custom directives from platform.conf: None
2021-12-16 13:16:14,077 Custom directives from platform.conf: None
2021-12-16 13:16:14,351 /esarchive/scratch/Earth/wuruchi/a29z/LOG_a29z has been created on bscearth000 .
2021-12-16 13:16:14,366 Submitting
2021-12-16 13:16:14,371 a29z_19550501_fc5_INI_STAT have been removed
2021-12-16 13:16:14,374 a29z_19550501_fc5_INI_COMPLETED been removed
2021-12-16 13:16:14,884 Job ID: 10209

2021-12-16 13:16:14,884 a29z_19550501_fc5_INI submitted
2021-12-16 13:16:15,047 
Jobs ready for marenostrum4: 3
2021-12-16 13:16:15,049 Number of jobs available: 3
2021-12-16 13:16:15,049 Number of jobs ready: 3
2021-12-16 13:16:15,050 Jobs ready for marenostrum4: 3
2021-12-16 13:16:15,056 Creating Scripts
2021-12-16 13:16:15,068 Sending Files
2021-12-16 13:16:15,129 Submitting
2021-12-16 13:16:15,136 a29z_19550501_fc0_1_SIM_STAT have been removed
2021-12-16 13:16:15,147 a29z_19550501_fc0_1_SIM_COMPLETED been removed
2021-12-16 13:16:15,150 Creating Scripts
2021-12-16 13:16:15,162 Sending Files
2021-12-16 13:16:15,238 Submitting
2021-12-16 13:16:15,241 a29z_19550501_fc1_1_SIM_STAT have been removed
2021-12-16 13:16:15,244 a29z_19550501_fc1_1_SIM_COMPLETED been removed
2021-12-16 13:16:15,246 Creating Scripts
2021-12-16 13:16:15,259 Sending Files
2021-12-16 13:16:15,274 Submitting
2021-12-16 13:16:15,277 a29z_19550501_fc2_1_SIM_STAT have been removed
2021-12-16 13:16:15,279 a29z_19550501_fc2_1_SIM_COMPLETED been removed
2021-12-16 13:16:15,335 Autosubmit will interrupt at the next safe occasion
[WARNING] 2021-12-16 13:16:16,560 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-16 13:16:16,563 Custom directives from platform.conf: None
2021-12-16 13:16:16,563 Custom directives from platform.conf: None
2021-12-16 13:16:21,177 Updating FAILED jobs
2021-12-16 13:16:21,178 Updating WAITING jobs
2021-12-16 13:16:21,179 Update finished
2021-12-16 13:16:21,184 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-16 13:16:21,185 a29z_LOCAL_SETUP                   7696           COMPLETED      transfer_node       serial         
2021-12-16 13:16:21,185 a29z_REMOTE_SETUP                  8214           COMPLETED      transfer_node       serial         
2021-12-16 13:16:21,185 a29z_19550501_fc0_INI              8560           COMPLETED      transfer_node       serial         
2021-12-16 13:16:21,185 a29z_19550501_fc1_INI              8737           COMPLETED      transfer_node       serial         
2021-12-16 13:16:21,185 a29z_19550501_fc2_INI              8920           COMPLETED      transfer_node       serial         
2021-12-16 13:16:21,185 a29z_19550501_fc3_INI              9774           SUBMITTED      transfer_node       serial         
2021-12-16 13:16:21,185 a29z_19550501_fc4_INI              9960           SUBMITTED      transfer_node       serial         
2021-12-16 13:16:21,185 a29z_19550501_fc5_INI              10209          SUBMITTED      transfer_node       serial         
2021-12-16 13:16:21,185 a29z_19550501_fc0_1_SIM            19178169       SUBMITTED      marenostrum4        debug          
2021-12-16 13:16:21,185 a29z_19550501_fc1_1_SIM            19178170       SUBMITTED      marenostrum4        debug          
2021-12-16 13:16:21,185 a29z_19550501_fc2_1_SIM            19178171       SUBMITTED      marenostrum4        debug          
2021-12-16 13:16:21,188 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-16 13:16:21,196 Job list saved
2021-12-16 13:16:21,258 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-16 13:16:21,258 a29z_LOCAL_SETUP                   7696           COMPLETED      transfer_node       serial         
2021-12-16 13:16:21,258 a29z_REMOTE_SETUP                  8214           COMPLETED      transfer_node       serial         
2021-12-16 13:16:21,259 a29z_19550501_fc0_INI              8560           COMPLETED      transfer_node       serial         
2021-12-16 13:16:21,259 a29z_19550501_fc1_INI              8737           COMPLETED      transfer_node       serial         
2021-12-16 13:16:21,259 a29z_19550501_fc2_INI              8920           COMPLETED      transfer_node       serial         
2021-12-16 13:16:21,259 a29z_19550501_fc3_INI              9774           SUBMITTED      transfer_node       serial         
2021-12-16 13:16:21,259 a29z_19550501_fc4_INI              9960           SUBMITTED      transfer_node       serial         
2021-12-16 13:16:21,259 a29z_19550501_fc5_INI              10209          SUBMITTED      transfer_node       serial         
2021-12-16 13:16:21,259 a29z_19550501_fc0_1_SIM            19178169       SUBMITTED      marenostrum4        debug          
2021-12-16 13:16:21,259 a29z_19550501_fc1_1_SIM            19178170       SUBMITTED      marenostrum4        debug          
2021-12-16 13:16:21,259 a29z_19550501_fc2_1_SIM            19178171       SUBMITTED      marenostrum4        debug          
2021-12-16 13:16:21,260 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-16 13:16:21,269 Job list saved
2021-12-16 13:16:31,283 Looking for active threads before closing Autosubmit. Ending the program before these threads finish may result in unexpected behavior. This procedure will last until all threads have finished or the program has waited for more than 30 seconds.
