2021-12-29 21:27:45,681 
Checking configuration files...
[WARNING] 2021-12-29 21:27:45,695 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:27:45,697 expdef_a29z.conf OK
2021-12-29 21:27:45,697 platforms_a29z.conf OK
2021-12-29 21:27:45,698 jobs_a29z.conf OK
2021-12-29 21:27:45,700 wrappers OK
2021-12-29 21:27:45,700 autosubmit_a29z.conf OK
2021-12-29 21:27:45,700 Configuration files OK

2021-12-29 21:27:45,700 Autosubmit is running with 3.13.0
2021-12-29 21:27:45,703 Preparing .lock file to avoid multiple instances with same experiment id
2021-12-29 21:27:45,721 Custom directives from platform.conf: None
2021-12-29 21:27:45,721 Custom directives from platform.conf: None
2021-12-29 21:27:45,722 The Experiment name is: a29z
2021-12-29 21:27:45,722 Sleep: 10
2021-12-29 21:27:45,722 Default retrials: 0
2021-12-29 21:27:45,722 Starting job submission...
2021-12-29 21:27:45,753 Creating jobs...
2021-12-29 21:27:45,754 Loading JobList
2021-12-29 21:27:45,824 Creating LOCAL_SETUP jobs
2021-12-29 21:27:45,825 Creating REMOTE_SETUP jobs
2021-12-29 21:27:45,825 Creating INI jobs
2021-12-29 21:27:45,831 Creating SIM jobs
2021-12-29 21:27:45,870 Creating POST jobs
2021-12-29 21:27:45,905 Creating CLEAN jobs
2021-12-29 21:27:45,941 Creating TRANSFER jobs
2021-12-29 21:27:45,950 Creating LOCALPOST jobs
2021-12-29 21:27:45,953 Creating PLOT jobs
2021-12-29 21:27:45,956 Adding dependencies...
2021-12-29 21:27:45,956 Adding dependencies for LOCAL_SETUP jobs
2021-12-29 21:27:45,957 Adding dependencies for REMOTE_SETUP jobs
2021-12-29 21:27:45,957 Adding dependencies for INI jobs
2021-12-29 21:27:45,957 Adding dependencies for SIM jobs
2021-12-29 21:27:45,970 Adding dependencies for POST jobs
2021-12-29 21:27:45,975 Adding dependencies for CLEAN jobs
2021-12-29 21:27:45,980 Adding dependencies for TRANSFER jobs
2021-12-29 21:27:45,984 Adding dependencies for LOCALPOST jobs
2021-12-29 21:27:45,985 Adding dependencies for PLOT jobs
2021-12-29 21:27:45,986 Removing redundant dependencies...
2021-12-29 21:27:46,068 Updating structure persistence...
2021-12-29 21:27:46,252 Starting from job list restored from /esarchive/autosubmit/a29z/pkl files
2021-12-29 21:27:46,252 Length of the jobs list: 590
2021-12-29 21:27:46,253 Loading parameters...
2021-12-29 21:27:46,254 Checking experiment templates...
2021-12-29 21:27:46,257 Checking scripts...
2021-12-29 21:27:46,258 Wallclock for a29z_LOCAL_SETUP is not defined! , setting it to 02:00
2021-12-29 21:27:47,527 147 of 590 checked
2021-12-29 21:27:48,748 294 of 590 checked
2021-12-29 21:27:49,966 441 of 590 checked
2021-12-29 21:27:50,630 Wallclock for a29z_19550501_fc0_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,640 Wallclock for a29z_19550501_fc0_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,648 Wallclock for a29z_19550501_fc0_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,656 Wallclock for a29z_19550501_fc1_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,664 Wallclock for a29z_19550501_fc1_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,672 Wallclock for a29z_19550501_fc1_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,681 Wallclock for a29z_19550501_fc2_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,689 Wallclock for a29z_19550501_fc2_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,698 Wallclock for a29z_19550501_fc2_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,706 Wallclock for a29z_19550501_fc3_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,714 Wallclock for a29z_19550501_fc3_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,722 Wallclock for a29z_19550501_fc3_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,730 Wallclock for a29z_19550501_fc4_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,739 Wallclock for a29z_19550501_fc4_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,750 Wallclock for a29z_19550501_fc4_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,758 Wallclock for a29z_19550501_fc5_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,767 Wallclock for a29z_19550501_fc5_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,775 Wallclock for a29z_19550501_fc5_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,783 Wallclock for a29z_19550501_fc6_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,792 Wallclock for a29z_19550501_fc6_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,800 Wallclock for a29z_19550501_fc6_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,809 Wallclock for a29z_19551101_fc0_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,818 Wallclock for a29z_19551101_fc0_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,829 Wallclock for a29z_19551101_fc0_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,837 Wallclock for a29z_19551101_fc1_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,846 Wallclock for a29z_19551101_fc1_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,854 Wallclock for a29z_19551101_fc1_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,862 Wallclock for a29z_19551101_fc2_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,871 Wallclock for a29z_19551101_fc2_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,879 Wallclock for a29z_19551101_fc2_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,888 Wallclock for a29z_19551101_fc3_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,896 Wallclock for a29z_19551101_fc3_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,904 Wallclock for a29z_19551101_fc3_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,912 Wallclock for a29z_19551101_fc4_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,921 Wallclock for a29z_19551101_fc4_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,928 Wallclock for a29z_19551101_fc4_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,937 Wallclock for a29z_19551101_fc5_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,945 Wallclock for a29z_19551101_fc5_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,953 Wallclock for a29z_19551101_fc5_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,961 Wallclock for a29z_19551101_fc6_5_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,970 Wallclock for a29z_19551101_fc6_10_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:50,978 Wallclock for a29z_19551101_fc6_12_TRANSFER is not defined! , setting it to 02:00
2021-12-29 21:27:51,191 588 of 590 checked
2021-12-29 21:27:51,207 590 of 590 checked
2021-12-29 21:27:51,216 Scripts OK
2021-12-29 21:27:51,216 Loading job packages
2021-12-29 21:27:51,250 Checking job_list current status
2021-12-29 21:27:51,251 Updating FAILED jobs
2021-12-29 21:27:51,252 Updating WAITING jobs
2021-12-29 21:27:51,254 Updating WAITING jobs eligible for be prepared
2021-12-29 21:27:51,258 Updating Held jobs
2021-12-29 21:27:51,258 Update finished
2021-12-29 21:27:51,260 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-29 21:27:51,260 a29z_19550501_fc0_1_CLEAN          19588000       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:51,260 a29z_19550501_fc1_1_CLEAN          19588013       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:51,260 a29z_19550501_fc2_1_CLEAN          19588021       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:51,260 a29z_19550501_fc3_1_CLEAN          19588028       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:51,261 a29z_19550501_fc4_1_CLEAN          19588048       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:51,261 a29z_19550501_fc3_2_SIM            19588082       RUNNING        marenostrum4        debug          
2021-12-29 21:27:51,261 a29z_19550501_fc0_3_SIM            19588112       QUEUING        marenostrum4        debug          
2021-12-29 21:27:51,261 a29z_19550501_fc0_2_CLEAN          19588091       QUEUING        marenostrum4        debug          
2021-12-29 21:27:51,261 a29z_19550501_fc1_2_CLEAN          19588117       QUEUING        marenostrum4        debug          
2021-12-29 21:27:51,264 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-29 21:27:51,305 Job list saved
2021-12-29 21:27:51,307 Autosubmit is running with v3.13.0
2021-12-29 21:27:51,308 Running job data structure
2021-12-29 21:27:51,535 Checking the connection to all platforms in use
2021-12-29 21:27:52,959 [marenostrum4] Connection successful to host mn1.bsc.es
2021-12-29 21:27:52,963 [marenostrum4] Correct user privileges for host mn1.bsc.es
2021-12-29 21:27:53,626 [transfer_node] Connection successful to host bscearth000
2021-12-29 21:27:53,635 [transfer_node] Correct user privileges for host bscearth000
2021-12-29 21:27:53,635 Running main loop
2021-12-29 21:27:53,637 Reloading parameters...
[WARNING] 2021-12-29 21:27:53,647 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:27:53,648 Loading parameters...
2021-12-29 21:27:53,650 

553 of 590 jobs remaining (21:27)
2021-12-29 21:27:53,651 Sleep: 10
2021-12-29 21:27:53,651 Number of retrials: 0
2021-12-29 21:27:53,651 WRAPPER CHECK TIME = 10
2021-12-29 21:27:53,654 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z_backup.pkl
2021-12-29 21:27:53,692 Job list saved
2021-12-29 21:27:53,705 Checking jobs for platform=marenostrum4
2021-12-29 21:27:53,716 Checking jobs for platform=transfer_node
2021-12-29 21:27:53,716 Checking all jobs at once
2021-12-29 21:27:54,884 Successful check job command
2021-12-29 21:27:54,885 a29z_19550501_fc0_3_SIM job seems to have completed: checking...
2021-12-29 21:27:54,899 Job a29z_19550501_fc0_3_SIM is COMPLETED
2021-12-29 21:27:54,912 a29z_19550501_fc0_3_SIM_STAT file have been transfered
2021-12-29 21:27:54,988 a29z_19550501_fc0_3_SIM_STAT file have been transfered
[WARNING] 2021-12-29 21:27:55,049 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:27:55,051 a29z_19550501_fc3_2_SIM job seems to have completed: checking...
[WARNING] 2021-12-29 21:27:55,063 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:27:55,064 Job a29z_19550501_fc3_2_SIM is COMPLETED
2021-12-29 21:27:55,067 Custom directives from platform.conf: None
2021-12-29 21:27:55,068 Custom directives from platform.conf: None
2021-12-29 21:27:55,078 a29z_19550501_fc3_2_SIM_STAT file have been transfered
[WARNING] 2021-12-29 21:27:55,240 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:27:55,242 a29z_19550501_fc1_2_CLEAN job seems to have completed: checking...
[WARNING] 2021-12-29 21:27:55,254 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:27:55,256 Job a29z_19550501_fc1_2_CLEAN is COMPLETED
2021-12-29 21:27:55,258 Custom directives from platform.conf: None
2021-12-29 21:27:55,258 Custom directives from platform.conf: None
2021-12-29 21:27:55,342 a29z_19550501_fc1_2_CLEAN_STAT file have been transfered
2021-12-29 21:27:55,461 a29z_19550501_fc1_2_CLEAN_STAT file have been transfered
[WARNING] 2021-12-29 21:27:55,559 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:27:55,560 a29z_19550501_fc0_2_CLEAN job seems to have completed: checking...
2021-12-29 21:27:55,573 Job a29z_19550501_fc0_2_CLEAN is COMPLETED
[WARNING] 2021-12-29 21:27:55,573 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:27:55,576 Custom directives from platform.conf: None
2021-12-29 21:27:55,576 Custom directives from platform.conf: None
2021-12-29 21:27:55,585 a29z_19550501_fc0_2_CLEAN_STAT file have been transfered
2021-12-29 21:27:55,808 a29z_19550501_fc0_2_CLEAN_STAT file have been transfered
[WARNING] 2021-12-29 21:27:55,878 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:27:55,880 End of checking
2021-12-29 21:27:55,881 Updating FAILED jobs
2021-12-29 21:27:55,882 Updating WAITING jobs
2021-12-29 21:27:55,883 Setting job: a29z_19550501_fc0_3_POST status to: READY (all parents completed)...
2021-12-29 21:27:55,883 Setting job: a29z_19550501_fc3_2_POST status to: READY (all parents completed)...
2021-12-29 21:27:55,886 Resetting job: a29z_19550501_fc0_4_SIM status to: READY for retrial...
2021-12-29 21:27:55,887 A job in prepared status has all parent completed, job: a29z_19550501_fc0_4_SIM status set to: READY ...
2021-12-29 21:27:55,887 Resetting job: a29z_19550501_fc3_3_SIM status to: READY for retrial...
2021-12-29 21:27:55,887 A job in prepared status has all parent completed, job: a29z_19550501_fc3_3_SIM status set to: READY ...
2021-12-29 21:27:55,887 Updating WAITING jobs eligible for be prepared
2021-12-29 21:27:55,891 Updating Held jobs
2021-12-29 21:27:55,892 Update finished
2021-12-29 21:27:55,894 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-29 21:27:55,894 a29z_19550501_fc1_1_CLEAN          19588013       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:55,894 a29z_19550501_fc1_2_CLEAN          19588117       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:55,894 a29z_19550501_fc2_1_CLEAN          19588021       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:55,894 a29z_19550501_fc3_1_CLEAN          19588028       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:55,894 a29z_19550501_fc4_1_CLEAN          19588048       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:55,897 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
[WARNING] 2021-12-29 21:27:55,906 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:27:55,910 Custom directives from platform.conf: None
2021-12-29 21:27:55,911 Custom directives from platform.conf: None
2021-12-29 21:27:55,981 Job list saved
2021-12-29 21:27:55,986 
Jobs ready for marenostrum4: 15
2021-12-29 21:27:55,995 Number of jobs available: 3
2021-12-29 21:27:55,998 Number of jobs ready: 15
2021-12-29 21:27:56,006 Jobs ready for marenostrum4: 15
2021-12-29 21:27:56,082 Creating Scripts
2021-12-29 21:27:56,211 Sending Files
2021-12-29 21:27:56,221 Submitting
2021-12-29 21:27:56,229 a29z_19550501_fc0_4_SIM_STAT have been removed
2021-12-29 21:27:56,231 a29z_19550501_fc0_4_SIM_COMPLETED been removed
2021-12-29 21:27:56,263 Creating Scripts
2021-12-29 21:27:56,310 Sending Files
2021-12-29 21:27:56,320 Submitting
2021-12-29 21:27:56,323 a29z_19550501_fc1_3_SIM_STAT have been removed
2021-12-29 21:27:56,325 a29z_19550501_fc1_3_SIM_COMPLETED been removed
2021-12-29 21:27:56,333 Creating Scripts
2021-12-29 21:27:56,383 Sending Files
2021-12-29 21:27:56,392 Submitting
2021-12-29 21:27:56,399 a29z_19550501_fc2_2_CLEAN_STAT have been removed
2021-12-29 21:27:56,401 a29z_19550501_fc2_2_CLEAN_COMPLETED been removed
2021-12-29 21:27:57,788 
Jobs ready for transfer_node: 2
2021-12-29 21:27:57,798 Number of jobs available: 3
2021-12-29 21:27:57,801 Number of jobs ready: 2
2021-12-29 21:27:57,806 Jobs ready for transfer_node: 2
2021-12-29 21:27:57,816 Creating Scripts
2021-12-29 21:27:57,856 Sending Files
2021-12-29 21:27:58,124 /esarchive/scratch/Earth/wuruchi/a29z/LOG_a29z has been created on bscearth000 .
2021-12-29 21:27:58,138 Submitting
2021-12-29 21:27:58,147 a29z_19550501_fc0_3_POST_STAT have been removed
2021-12-29 21:27:58,150 a29z_19550501_fc0_3_POST_COMPLETED been removed
2021-12-29 21:27:58,418 Job ID: 6654

2021-12-29 21:27:58,418 a29z_19550501_fc0_3_POST submitted
2021-12-29 21:27:58,535 Creating Scripts
2021-12-29 21:27:58,578 Sending Files
2021-12-29 21:27:58,896 /esarchive/scratch/Earth/wuruchi/a29z/LOG_a29z has been created on bscearth000 .
2021-12-29 21:27:58,908 Submitting
2021-12-29 21:27:59,492 Job ID: 6828

2021-12-29 21:27:59,492 a29z_19550501_fc3_2_POST submitted
2021-12-29 21:27:59,576 Updating FAILED jobs
2021-12-29 21:27:59,577 Updating WAITING jobs
2021-12-29 21:27:59,580 Updating WAITING jobs eligible for be prepared
2021-12-29 21:27:59,583 Updating Held jobs
2021-12-29 21:27:59,584 Update finished
2021-12-29 21:27:59,585 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-29 21:27:59,585 a29z_19550501_fc1_1_CLEAN          19588013       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:59,586 a29z_19550501_fc1_2_CLEAN          19588117       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:59,586 a29z_19550501_fc2_1_CLEAN          19588021       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:59,586 a29z_19550501_fc3_1_CLEAN          19588028       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:59,586 a29z_19550501_fc4_1_CLEAN          19588048       COMPLETED      marenostrum4        debug          
2021-12-29 21:27:59,586 a29z_19550501_fc0_4_SIM            19598040       SUBMITTED      marenostrum4        debug          
2021-12-29 21:27:59,586 a29z_19550501_fc1_3_SIM            19598041       SUBMITTED      marenostrum4        debug          
2021-12-29 21:27:59,586 a29z_19550501_fc0_3_POST           6654           SUBMITTED      transfer_node       serial         
2021-12-29 21:27:59,587 a29z_19550501_fc3_2_POST           6828           SUBMITTED      transfer_node       serial         
2021-12-29 21:27:59,587 a29z_19550501_fc2_2_CLEAN          19598042       SUBMITTED      marenostrum4        debug          
2021-12-29 21:27:59,590 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-29 21:27:59,631 Job list saved
2021-12-29 21:27:59,705 Updating FAILED jobs
2021-12-29 21:27:59,706 Updating WAITING jobs
2021-12-29 21:27:59,709 Updating WAITING jobs eligible for be prepared
2021-12-29 21:27:59,712 Updating Held jobs
2021-12-29 21:27:59,712 Update finished
2021-12-29 21:28:09,717 Reloading parameters...
[WARNING] 2021-12-29 21:28:09,728 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:28:09,729 Loading parameters...
2021-12-29 21:28:09,731 

549 of 590 jobs remaining (21:28)
2021-12-29 21:28:09,731 Sleep: 10
2021-12-29 21:28:09,732 Number of retrials: 0
2021-12-29 21:28:09,732 WRAPPER CHECK TIME = 10
2021-12-29 21:28:09,743 Checking jobs for platform=marenostrum4
2021-12-29 21:28:09,754 Checking jobs for platform=transfer_node
2021-12-29 21:28:12,024 Successful check job command: nohup kill -0 6828 > /dev/null 2>&1; echo $?
2021-12-29 21:28:12,024 Job a29z_19550501_fc3_2_POST is RUNNING
2021-12-29 21:28:12,041 a29z_19550501_fc3_2_POST_STAT file have been transfered
2021-12-29 21:28:14,369 Successful check job command: nohup kill -0 6654 > /dev/null 2>&1; echo $?
2021-12-29 21:28:14,370 Job a29z_19550501_fc0_3_POST is RUNNING
2021-12-29 21:28:14,398 a29z_19550501_fc0_3_POST_STAT file have been transfered
2021-12-29 21:28:14,447 Checking all jobs at once
2021-12-29 21:28:15,720 Successful check job command
2021-12-29 21:28:17,341 Job a29z_19550501_fc0_4_SIM is RUNNING
2021-12-29 21:28:17,360 a29z_19550501_fc0_4_SIM_STAT file have been transfered
2021-12-29 21:28:17,415 Job a29z_19550501_fc1_3_SIM is QUEUING
2021-12-29 21:28:17,415 Job a29z_19550501_fc2_2_CLEAN is QUEUING
2021-12-29 21:28:17,415 End of checking
2021-12-29 21:28:17,417 Updating FAILED jobs
2021-12-29 21:28:17,417 Updating WAITING jobs
2021-12-29 21:28:17,420 Updating WAITING jobs eligible for be prepared
2021-12-29 21:28:17,422 Setting job: a29z_19550501_fc1_4_SIM status to: Prepared for be held (all parents queuing, running or completed)...
2021-12-29 21:28:17,423 Setting job: a29z_19550501_fc0_3_CLEAN status to: Prepared for be held (all parents queuing, running or completed)...
2021-12-29 21:28:17,423 Setting job: a29z_19550501_fc3_2_CLEAN status to: Prepared for be held (all parents queuing, running or completed)...
2021-12-29 21:28:17,424 Updating Held jobs
2021-12-29 21:28:17,424 Update finished
2021-12-29 21:28:17,426 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-29 21:28:17,426 a29z_19550501_fc1_1_CLEAN          19588013       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:17,427 a29z_19550501_fc1_2_CLEAN          19588117       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:17,427 a29z_19550501_fc2_1_CLEAN          19588021       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:17,427 a29z_19550501_fc3_1_CLEAN          19588028       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:17,427 a29z_19550501_fc4_1_CLEAN          19588048       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:17,427 a29z_19550501_fc0_4_SIM            19598040       RUNNING        marenostrum4        debug          
2021-12-29 21:28:17,427 a29z_19550501_fc0_3_POST           6654           RUNNING        transfer_node       serial         
2021-12-29 21:28:17,427 a29z_19550501_fc3_2_POST           6828           RUNNING        transfer_node       serial         
2021-12-29 21:28:17,428 a29z_19550501_fc1_3_SIM            19598041       QUEUING        marenostrum4        debug          
2021-12-29 21:28:17,428 a29z_19550501_fc2_2_CLEAN          19598042       QUEUING        marenostrum4        debug          
2021-12-29 21:28:17,430 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-29 21:28:17,469 Job list saved
2021-12-29 21:28:17,474 
Jobs ready for marenostrum4: 12
2021-12-29 21:28:17,483 Number of jobs available: 1
2021-12-29 21:28:17,485 Number of jobs ready: 12
2021-12-29 21:28:17,490 Jobs ready for marenostrum4: 12
2021-12-29 21:28:17,505 Creating Scripts
2021-12-29 21:28:17,546 Sending Files
2021-12-29 21:28:17,558 Submitting
2021-12-29 21:28:17,562 a29z_19550501_fc2_3_SIM_STAT have been removed
2021-12-29 21:28:17,563 a29z_19550501_fc2_3_SIM_COMPLETED been removed
2021-12-29 21:28:18,805 
Jobs ready for transfer_node: 0
2021-12-29 21:28:18,816 Number of jobs available: 3
2021-12-29 21:28:18,818 Number of jobs ready: 0
2021-12-29 21:28:18,823 Updating FAILED jobs
2021-12-29 21:28:18,824 Updating WAITING jobs
2021-12-29 21:28:18,827 Updating WAITING jobs eligible for be prepared
2021-12-29 21:28:18,831 Updating Held jobs
2021-12-29 21:28:18,831 Update finished
2021-12-29 21:28:18,833 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-29 21:28:18,833 a29z_19550501_fc1_1_CLEAN          19588013       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:18,833 a29z_19550501_fc1_2_CLEAN          19588117       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:18,833 a29z_19550501_fc2_1_CLEAN          19588021       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:18,833 a29z_19550501_fc3_1_CLEAN          19588028       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:18,833 a29z_19550501_fc4_1_CLEAN          19588048       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:18,834 a29z_19550501_fc2_3_SIM            19598043       SUBMITTED      marenostrum4        debug          
2021-12-29 21:28:18,834 a29z_19550501_fc0_4_SIM            19598040       RUNNING        marenostrum4        debug          
2021-12-29 21:28:18,834 a29z_19550501_fc0_3_POST           6654           RUNNING        transfer_node       serial         
2021-12-29 21:28:18,834 a29z_19550501_fc3_2_POST           6828           RUNNING        transfer_node       serial         
2021-12-29 21:28:18,834 a29z_19550501_fc1_3_SIM            19598041       QUEUING        marenostrum4        debug          
2021-12-29 21:28:18,834 a29z_19550501_fc2_2_CLEAN          19598042       QUEUING        marenostrum4        debug          
2021-12-29 21:28:18,836 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-29 21:28:18,875 Job list saved
2021-12-29 21:28:18,880 
Jobs prepared for marenostrum4: 3
2021-12-29 21:28:18,886 Number of jobs available: 0
2021-12-29 21:28:18,889 Number of jobs prepared: 3
2021-12-29 21:28:18,894 Jobs ready for marenostrum4: 3
2021-12-29 21:28:18,904 
Jobs prepared for transfer_node: 0
2021-12-29 21:28:18,910 Number of jobs available: 3
2021-12-29 21:28:18,912 Number of jobs prepared: 0
2021-12-29 21:28:18,918 Updating FAILED jobs
2021-12-29 21:28:18,919 Updating WAITING jobs
2021-12-29 21:28:18,921 Updating WAITING jobs eligible for be prepared
2021-12-29 21:28:18,924 Updating Held jobs
2021-12-29 21:28:18,925 Update finished
2021-12-29 21:28:18,926 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-29 21:28:18,926 a29z_19550501_fc1_1_CLEAN          19588013       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:18,927 a29z_19550501_fc1_2_CLEAN          19588117       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:18,927 a29z_19550501_fc2_1_CLEAN          19588021       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:18,927 a29z_19550501_fc3_1_CLEAN          19588028       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:18,927 a29z_19550501_fc4_1_CLEAN          19588048       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:18,927 a29z_19550501_fc2_3_SIM            19598043       SUBMITTED      marenostrum4        debug          
2021-12-29 21:28:18,927 a29z_19550501_fc0_4_SIM            19598040       RUNNING        marenostrum4        debug          
2021-12-29 21:28:18,927 a29z_19550501_fc0_3_POST           6654           RUNNING        transfer_node       serial         
2021-12-29 21:28:18,927 a29z_19550501_fc3_2_POST           6828           RUNNING        transfer_node       serial         
2021-12-29 21:28:18,928 a29z_19550501_fc1_3_SIM            19598041       QUEUING        marenostrum4        debug          
2021-12-29 21:28:18,928 a29z_19550501_fc2_2_CLEAN          19598042       QUEUING        marenostrum4        debug          
2021-12-29 21:28:18,930 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-29 21:28:18,968 Job list saved
2021-12-29 21:28:19,018 Updating FAILED jobs
2021-12-29 21:28:19,019 Updating WAITING jobs
2021-12-29 21:28:19,022 Updating WAITING jobs eligible for be prepared
2021-12-29 21:28:19,025 Updating Held jobs
2021-12-29 21:28:19,025 Update finished
2021-12-29 21:28:29,033 Reloading parameters...
[WARNING] 2021-12-29 21:28:29,044 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:28:29,045 Loading parameters...
2021-12-29 21:28:29,048 

549 of 590 jobs remaining (21:28)
2021-12-29 21:28:29,048 Sleep: 10
2021-12-29 21:28:29,048 Number of retrials: 0
2021-12-29 21:28:29,048 WRAPPER CHECK TIME = 10
2021-12-29 21:28:29,059 Checking jobs for platform=marenostrum4
2021-12-29 21:28:29,070 Checking jobs for platform=transfer_node
2021-12-29 21:28:31,340 Successful check job command: nohup kill -0 6828 > /dev/null 2>&1; echo $?
2021-12-29 21:28:31,340 a29z_19550501_fc3_2_POST job seems to have completed: checking...
2021-12-29 21:28:31,369 Job a29z_19550501_fc3_2_POST is COMPLETED
2021-12-29 21:28:31,389 a29z_19550501_fc3_2_POST_STAT file have been transfered
[WARNING] 2021-12-29 21:28:31,456 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
[WARNING] 2021-12-29 21:28:31,469 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:28:31,473 Custom directives from platform.conf: None
2021-12-29 21:28:31,473 Custom directives from platform.conf: None
2021-12-29 21:28:33,677 Successful check job command: nohup kill -0 6654 > /dev/null 2>&1; echo $?
2021-12-29 21:28:33,677 a29z_19550501_fc0_3_POST job seems to have completed: checking...
2021-12-29 21:28:34,567 Job a29z_19550501_fc0_3_POST is COMPLETED
2021-12-29 21:28:34,864 a29z_19550501_fc0_3_POST_STAT file have been transfered
[WARNING] 2021-12-29 21:28:34,928 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:28:34,929 Checking all jobs at once
[WARNING] 2021-12-29 21:28:34,941 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:28:34,944 Custom directives from platform.conf: None
2021-12-29 21:28:34,945 Custom directives from platform.conf: None
2021-12-29 21:28:36,095 Successful check job command
2021-12-29 21:28:37,314 a29z_19550501_fc0_4_SIM job seems to have completed: checking...
2021-12-29 21:28:37,334 Job a29z_19550501_fc0_4_SIM is COMPLETED
2021-12-29 21:28:37,375 a29z_19550501_fc0_4_SIM_STAT file have been transfered
[WARNING] 2021-12-29 21:28:37,446 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:28:37,448 Job a29z_19550501_fc1_3_SIM is QUEUING
2021-12-29 21:28:37,448 Job a29z_19550501_fc2_2_CLEAN is QUEUING
2021-12-29 21:28:37,449 Job a29z_19550501_fc2_3_SIM is QUEUING
2021-12-29 21:28:37,449 End of checking
2021-12-29 21:28:37,450 Updating FAILED jobs
2021-12-29 21:28:37,451 Updating WAITING jobs
2021-12-29 21:28:37,453 Setting job: a29z_19550501_fc0_4_POST status to: READY (all parents completed)...
2021-12-29 21:28:37,455 Resetting job: a29z_19550501_fc0_3_CLEAN status to: READY for retrial...
2021-12-29 21:28:37,456 A job in prepared status has all parent completed, job: a29z_19550501_fc0_3_CLEAN status set to: READY ...
2021-12-29 21:28:37,456 Resetting job: a29z_19550501_fc3_2_CLEAN status to: READY for retrial...
2021-12-29 21:28:37,456 A job in prepared status has all parent completed, job: a29z_19550501_fc3_2_CLEAN status set to: READY ...
2021-12-29 21:28:37,457 Updating WAITING jobs eligible for be prepared
2021-12-29 21:28:37,459 Setting job: a29z_19550501_fc2_4_SIM status to: Prepared for be held (all parents queuing, running or completed)...
2021-12-29 21:28:37,461 Updating Held jobs
2021-12-29 21:28:37,462 Update finished
2021-12-29 21:28:37,464 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-29 21:28:37,465 a29z_19550501_fc1_1_CLEAN          19588013       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:37,465 a29z_19550501_fc1_2_CLEAN          19588117       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:37,466 a29z_19550501_fc2_1_CLEAN          19588021       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:37,466 a29z_19550501_fc3_1_CLEAN          19588028       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:37,467 a29z_19550501_fc4_1_CLEAN          19588048       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:37,467 a29z_19550501_fc1_3_SIM            19598041       QUEUING        marenostrum4        debug          
2021-12-29 21:28:37,467 a29z_19550501_fc2_3_SIM            19598043       QUEUING        marenostrum4        debug          
2021-12-29 21:28:37,467 a29z_19550501_fc2_2_CLEAN          19598042       QUEUING        marenostrum4        debug          
2021-12-29 21:28:37,470 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
[WARNING] 2021-12-29 21:28:37,477 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:28:37,482 Custom directives from platform.conf: None
2021-12-29 21:28:37,483 Custom directives from platform.conf: None
2021-12-29 21:28:37,553 Job list saved
2021-12-29 21:28:37,558 
Jobs ready for marenostrum4: 13
2021-12-29 21:28:37,567 Number of jobs available: 0
2021-12-29 21:28:37,570 Number of jobs ready: 13
2021-12-29 21:28:37,577 Jobs ready for marenostrum4: 13
2021-12-29 21:28:37,647 
Jobs ready for transfer_node: 1
2021-12-29 21:28:37,657 Number of jobs available: 3
2021-12-29 21:28:37,660 Number of jobs ready: 1
2021-12-29 21:28:37,665 Jobs ready for transfer_node: 1
2021-12-29 21:28:37,680 Creating Scripts
2021-12-29 21:28:37,749 Sending Files
2021-12-29 21:28:38,139 /esarchive/scratch/Earth/wuruchi/a29z/LOG_a29z has been created on bscearth000 .
2021-12-29 21:28:38,355 Submitting
2021-12-29 21:28:38,799 Job ID: 8053

2021-12-29 21:28:38,799 a29z_19550501_fc0_4_POST submitted
2021-12-29 21:28:38,886 Updating FAILED jobs
2021-12-29 21:28:38,887 Updating WAITING jobs
2021-12-29 21:28:38,890 Updating WAITING jobs eligible for be prepared
2021-12-29 21:28:38,893 Updating Held jobs
2021-12-29 21:28:38,893 Update finished
2021-12-29 21:28:38,895 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-29 21:28:38,895 a29z_19550501_fc1_1_CLEAN          19588013       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:38,895 a29z_19550501_fc1_2_CLEAN          19588117       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:38,895 a29z_19550501_fc2_1_CLEAN          19588021       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:38,895 a29z_19550501_fc3_1_CLEAN          19588028       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:38,895 a29z_19550501_fc4_1_CLEAN          19588048       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:38,896 a29z_19550501_fc0_4_POST           8053           SUBMITTED      transfer_node       serial         
2021-12-29 21:28:38,896 a29z_19550501_fc1_3_SIM            19598041       QUEUING        marenostrum4        debug          
2021-12-29 21:28:38,896 a29z_19550501_fc2_3_SIM            19598043       QUEUING        marenostrum4        debug          
2021-12-29 21:28:38,896 a29z_19550501_fc2_2_CLEAN          19598042       QUEUING        marenostrum4        debug          
2021-12-29 21:28:38,898 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-29 21:28:38,937 Job list saved
2021-12-29 21:28:38,941 
Jobs prepared for marenostrum4: 2
2021-12-29 21:28:38,949 Number of jobs available: 0
2021-12-29 21:28:38,951 Number of jobs prepared: 2
2021-12-29 21:28:38,955 Jobs ready for marenostrum4: 2
2021-12-29 21:28:38,965 
Jobs prepared for transfer_node: 0
2021-12-29 21:28:38,971 Number of jobs available: 2
2021-12-29 21:28:38,973 Number of jobs prepared: 0
2021-12-29 21:28:38,980 Updating FAILED jobs
2021-12-29 21:28:38,981 Updating WAITING jobs
2021-12-29 21:28:38,984 Updating WAITING jobs eligible for be prepared
2021-12-29 21:28:38,987 Updating Held jobs
2021-12-29 21:28:38,987 Update finished
2021-12-29 21:28:38,989 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-29 21:28:38,989 a29z_19550501_fc1_1_CLEAN          19588013       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:38,989 a29z_19550501_fc1_2_CLEAN          19588117       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:38,989 a29z_19550501_fc2_1_CLEAN          19588021       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:38,989 a29z_19550501_fc3_1_CLEAN          19588028       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:38,989 a29z_19550501_fc4_1_CLEAN          19588048       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:38,989 a29z_19550501_fc0_4_POST           8053           SUBMITTED      transfer_node       serial         
2021-12-29 21:28:38,990 a29z_19550501_fc1_3_SIM            19598041       QUEUING        marenostrum4        debug          
2021-12-29 21:28:38,990 a29z_19550501_fc2_3_SIM            19598043       QUEUING        marenostrum4        debug          
2021-12-29 21:28:38,990 a29z_19550501_fc2_2_CLEAN          19598042       QUEUING        marenostrum4        debug          
2021-12-29 21:28:38,992 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-29 21:28:39,030 Job list saved
2021-12-29 21:28:39,093 Updating FAILED jobs
2021-12-29 21:28:39,094 Updating WAITING jobs
2021-12-29 21:28:39,096 Updating WAITING jobs eligible for be prepared
2021-12-29 21:28:39,099 Updating Held jobs
2021-12-29 21:28:39,100 Update finished
2021-12-29 21:28:49,106 Reloading parameters...
[WARNING] 2021-12-29 21:28:49,120 /esarchive/autosubmit/a29z/conf/proj_a29z.conf was not found. Some variables might be missing. If your experiment does not need a proj file, you can ignore this message.
2021-12-29 21:28:49,121 Loading parameters...
2021-12-29 21:28:49,123 

546 of 590 jobs remaining (21:28)
2021-12-29 21:28:49,124 Sleep: 10
2021-12-29 21:28:49,124 Number of retrials: 0
2021-12-29 21:28:49,124 WRAPPER CHECK TIME = 10
2021-12-29 21:28:49,135 Checking jobs for platform=marenostrum4
2021-12-29 21:28:49,145 Checking jobs for platform=transfer_node
2021-12-29 21:28:51,622 Successful check job command: nohup kill -0 8053 > /dev/null 2>&1; echo $?
2021-12-29 21:28:51,622 Job a29z_19550501_fc0_4_POST is RUNNING
2021-12-29 21:28:51,840 a29z_19550501_fc0_4_POST_STAT file have been transfered
2021-12-29 21:28:51,894 Checking all jobs at once
2021-12-29 21:28:53,063 Successful check job command
2021-12-29 21:28:54,282 Job a29z_19550501_fc1_3_SIM is RUNNING
2021-12-29 21:28:54,296 a29z_19550501_fc1_3_SIM_STAT file have been transfered
2021-12-29 21:28:54,346 Job a29z_19550501_fc2_2_CLEAN is QUEUING
2021-12-29 21:28:54,347 Job a29z_19550501_fc2_3_SIM is QUEUING
2021-12-29 21:28:54,347 End of checking
2021-12-29 21:28:54,348 Updating FAILED jobs
2021-12-29 21:28:54,349 Updating WAITING jobs
2021-12-29 21:28:54,351 Updating WAITING jobs eligible for be prepared
2021-12-29 21:28:54,354 Setting job: a29z_19550501_fc0_4_CLEAN status to: Prepared for be held (all parents queuing, running or completed)...
2021-12-29 21:28:54,355 Updating Held jobs
2021-12-29 21:28:54,355 Update finished
2021-12-29 21:28:54,357 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-29 21:28:54,357 a29z_19550501_fc1_1_CLEAN          19588013       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:54,357 a29z_19550501_fc1_2_CLEAN          19588117       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:54,357 a29z_19550501_fc2_1_CLEAN          19588021       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:54,357 a29z_19550501_fc3_1_CLEAN          19588028       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:54,357 a29z_19550501_fc4_1_CLEAN          19588048       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:54,358 a29z_19550501_fc1_3_SIM            19598041       RUNNING        marenostrum4        debug          
2021-12-29 21:28:54,358 a29z_19550501_fc0_4_POST           8053           RUNNING        transfer_node       serial         
2021-12-29 21:28:54,358 a29z_19550501_fc2_3_SIM            19598043       QUEUING        marenostrum4        debug          
2021-12-29 21:28:54,358 a29z_19550501_fc2_2_CLEAN          19598042       QUEUING        marenostrum4        debug          
2021-12-29 21:28:54,361 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-29 21:28:54,400 Job list saved
2021-12-29 21:28:54,404 
Jobs ready for marenostrum4: 13
2021-12-29 21:28:54,413 Number of jobs available: 1
2021-12-29 21:28:54,415 Number of jobs ready: 13
2021-12-29 21:28:54,419 Jobs ready for marenostrum4: 13
2021-12-29 21:28:54,436 Creating Scripts
2021-12-29 21:28:54,476 Sending Files
2021-12-29 21:28:54,523 Submitting
2021-12-29 21:28:54,531 a29z_19550501_fc0_3_CLEAN_STAT have been removed
2021-12-29 21:28:54,532 a29z_19550501_fc0_3_CLEAN_COMPLETED been removed
2021-12-29 21:28:55,671 
Jobs ready for transfer_node: 0
2021-12-29 21:28:55,680 Number of jobs available: 3
2021-12-29 21:28:55,682 Number of jobs ready: 0
2021-12-29 21:28:55,688 Updating FAILED jobs
2021-12-29 21:28:55,688 Updating WAITING jobs
2021-12-29 21:28:55,691 Updating WAITING jobs eligible for be prepared
2021-12-29 21:28:55,694 Updating Held jobs
2021-12-29 21:28:55,694 Update finished
2021-12-29 21:28:55,696 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-29 21:28:55,697 a29z_19550501_fc1_1_CLEAN          19588013       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:55,697 a29z_19550501_fc1_2_CLEAN          19588117       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:55,697 a29z_19550501_fc2_1_CLEAN          19588021       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:55,697 a29z_19550501_fc3_1_CLEAN          19588028       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:55,697 a29z_19550501_fc4_1_CLEAN          19588048       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:55,697 a29z_19550501_fc0_3_CLEAN          19598046       SUBMITTED      marenostrum4        debug          
2021-12-29 21:28:55,697 a29z_19550501_fc1_3_SIM            19598041       RUNNING        marenostrum4        debug          
2021-12-29 21:28:55,698 a29z_19550501_fc0_4_POST           8053           RUNNING        transfer_node       serial         
2021-12-29 21:28:55,698 a29z_19550501_fc2_3_SIM            19598043       QUEUING        marenostrum4        debug          
2021-12-29 21:28:55,698 a29z_19550501_fc2_2_CLEAN          19598042       QUEUING        marenostrum4        debug          
2021-12-29 21:28:55,700 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-29 21:28:55,740 Job list saved
2021-12-29 21:28:55,744 
Jobs prepared for marenostrum4: 3
2021-12-29 21:28:55,752 Number of jobs available: 0
2021-12-29 21:28:55,754 Number of jobs prepared: 3
2021-12-29 21:28:55,759 Jobs ready for marenostrum4: 3
2021-12-29 21:28:55,770 
Jobs prepared for transfer_node: 0
2021-12-29 21:28:55,778 Number of jobs available: 3
2021-12-29 21:28:55,781 Number of jobs prepared: 0
2021-12-29 21:28:55,786 Updating FAILED jobs
2021-12-29 21:28:55,787 Updating WAITING jobs
2021-12-29 21:28:55,789 Updating WAITING jobs eligible for be prepared
2021-12-29 21:28:55,792 Updating Held jobs
2021-12-29 21:28:55,793 Update finished
2021-12-29 21:28:55,794 
Job Name                           Job Id         Job Status     Job Platform        Job Queue      
2021-12-29 21:28:55,794 a29z_19550501_fc1_1_CLEAN          19588013       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:55,795 a29z_19550501_fc1_2_CLEAN          19588117       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:55,795 a29z_19550501_fc2_1_CLEAN          19588021       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:55,795 a29z_19550501_fc3_1_CLEAN          19588028       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:55,795 a29z_19550501_fc4_1_CLEAN          19588048       COMPLETED      marenostrum4        debug          
2021-12-29 21:28:55,795 a29z_19550501_fc0_3_CLEAN          19598046       SUBMITTED      marenostrum4        debug          
2021-12-29 21:28:55,795 a29z_19550501_fc1_3_SIM            19598041       RUNNING        marenostrum4        debug          
2021-12-29 21:28:55,795 a29z_19550501_fc0_4_POST           8053           RUNNING        transfer_node       serial         
2021-12-29 21:28:55,795 a29z_19550501_fc2_3_SIM            19598043       QUEUING        marenostrum4        debug          
2021-12-29 21:28:55,796 a29z_19550501_fc2_2_CLEAN          19598042       QUEUING        marenostrum4        debug          
2021-12-29 21:28:55,798 Saving JobList: /esarchive/autosubmit/a29z/pkl/job_list_a29z.pkl
2021-12-29 21:28:55,836 Job list saved
2021-12-29 21:28:55,895 Updating FAILED jobs
2021-12-29 21:28:55,895 Updating WAITING jobs
2021-12-29 21:28:55,898 Updating WAITING jobs eligible for be prepared
2021-12-29 21:28:55,901 Updating Held jobs
2021-12-29 21:28:55,902 Update finished
2021-12-29 21:29:02,227 Autosubmit will interrupt at the next safe occasion
2021-12-29 21:29:02,229 Looking for active threads before closing Autosubmit. Ending the program before these threads finish may result in unexpected behavior. This procedure will last until all threads have finished or the program has waited for more than 30 seconds.
2021-12-29 21:29:02,229 JOB_data_a29z_19550501_fc0_4_SIM is still retrieving outputs, time remaining is 60 seconds.
2021-12-29 21:29:12,236 JOB_data_a29z_19550501_fc0_4_SIM is still retrieving outputs, time remaining is 50 seconds.
2021-12-29 21:29:22,247 JOB_data_a29z_19550501_fc0_4_SIM is still retrieving outputs, time remaining is 40 seconds.
2021-12-29 21:29:32,251 JOB_data_a29z_19550501_fc0_4_SIM is still retrieving outputs, time remaining is 30 seconds.
