testing version of autosubmit re-submits already running jobs
With the latest AS dev version, I was running an experiment and stopped AS to re-launch a failed job.
Then AS re-submitted a number of job which were already running (e.g. a1wj_19810101_fc03_1_SIM):
Job a1wj_19810101_fc03_1_SIM is RUNNING
Job a1wj_19810101_fc04_1_SIM is RUNNING
Job a1wj_19820101_fc00_1_SIM is RUNNING
Job a1wj_19820101_fc01_1_SIM is RUNNING
Job a1wj_19820101_fc02_1_SIM is RUNNING
Job a1wj_19820101_fc03_1_SIM is RUNNING
Job a1wj_19820101_fc04_1_SIM is RUNNING
Job a1wj_19830101_fc00_1_SIM is RUNNING
Job a1wj_19850101_fc02_1_SIM is RUNNING
Job a1wj_19910101_fc03_1_SIM is RUNNING
Job a1wj_19860101_fc04_1_SIM is RUNNING
Job a1wj_19870101_fc00_1_SIM is RUNNING
Job a1wj_19850101_fc01_1_SIM is RUNNING
Job a1wj_19900101_fc04_1_SIM is RUNNING
Job a1wj_19910101_fc00_1_SIM is RUNNING
Job a1wj_19920101_fc00_1_SIM is RUNNING
Job a1wj_19900101_fc00_1_SIM is RUNNING
Job a1wj_19900101_fc01_1_SIM is RUNNING
Job a1wj_19900101_fc02_1_SIM is RUNNING
Job a1wj_19890101_fc04_1_SIM is RUNNING
Job a1wj_19890101_fc01_1_SIM is RUNNING
Job a1wj_19850101_fc00_1_SIM is RUNNING
Job a1wj_19890101_fc03_1_SIM is RUNNING
Job a1wj_19890101_fc02_1_SIM is RUNNING
Jobs ready for marenostrum4: 24
631 of 723 jobs remaining (10:55)
[WARNING] Command sacct -n -X -j 5919538,5919590,5919571,5919545,5919533,5919532,5919530,5919519,5919518,5919535,5919534,5919515,5919514,5919517,5919516,5919559,5919558,5919542,5919551,5919550,5919553,5919552,5919555,5919554,5919557,5919556,5919546,5919567,5919547,5919544,5919573,5919543,5919537,5919536,5919520,5919521,5919522,5919523,5919524,5919525,5919526,5919527,5919528,5919529,5919541,5919572,5919548,5919549,5919540,5919568,5919569,5919585,5919564,5919565,5919566,5919563,5919560,5919539,5919562,5919561, -o 'jobid,State' in mn1.bsc.es warning: /home/cns54/cns54230/.bashrc: line 53: ppwd: command not found
Job a1wj_19840101_fc04_1_SIM is RUNNING
Job a1wj_19920101_fc01_1_SIM is RUNNING
Job a1wj_19910101_fc02_1_SIM is RUNNING
Job a1wj_19860101_fc01_1_SIM is RUNNING
Job a1wj_19830101_fc04_1_SIM is RUNNING
Job a1wj_19830101_fc03_1_SIM is RUNNING
Job a1wj_19830101_fc01_1_SIM is RUNNING
Job a1wj_19810101_fc00_1_SIM is RUNNING
Job a1wj_19800101_fc04_1_SIM is RUNNING
Job a1wj_19840101_fc01_1_SIM is RUNNING
Job a1wj_19840101_fc00_1_SIM is RUNNING
Job a1wj_19800101_fc01_1_SIM is RUNNING
Job a1wj_19800101_fc00_1_SIM is RUNNING
Job a1wj_19800101_fc03_1_SIM is RUNNING
Job a1wj_19800101_fc02_1_SIM is RUNNING
Job a1wj_19890101_fc00_1_SIM is RUNNING
Job a1wj_19880101_fc04_1_SIM is RUNNING
Job a1wj_19850101_fc03_1_SIM is RUNNING
Job a1wj_19870101_fc02_1_SIM is RUNNING
Job a1wj_19870101_fc01_1_SIM is RUNNING
Job a1wj_19870101_fc04_1_SIM is RUNNING
Job a1wj_19870101_fc03_1_SIM is RUNNING
Job a1wj_19880101_fc01_1_SIM is RUNNING
Job a1wj_19880101_fc00_1_SIM is RUNNING
Job a1wj_19880101_fc03_1_SIM is RUNNING
Job a1wj_19880101_fc02_1_SIM is RUNNING
Job a1wj_19860101_fc02_1_SIM is RUNNING
Job a1wj_19900101_fc03_1_SIM is RUNNING
Job a1wj_19860101_fc03_1_SIM is RUNNING
Job a1wj_19860101_fc00_1_SIM is RUNNING
Job a1wj_19910101_fc04_1_SIM is RUNNING
Job a1wj_19850101_fc04_1_SIM is RUNNING
Job a1wj_19840101_fc03_1_SIM is RUNNING
Job a1wj_19840101_fc02_1_SIM is RUNNING
Job a1wj_19810101_fc01_1_SIM is RUNNING
Job a1wj_19810101_fc02_1_SIM is RUNNING
Job a1wj_19810101_fc03_1_SIM is RUNNING
Job a1wj_19810101_fc04_1_SIM is RUNNING
Job a1wj_19820101_fc00_1_SIM is RUNNING
Job a1wj_19820101_fc01_1_SIM is RUNNING
Job a1wj_19820101_fc02_1_SIM is RUNNING
Job a1wj_19820101_fc03_1_SIM is RUNNING
Job a1wj_19820101_fc04_1_SIM is RUNNING
Job a1wj_19830101_fc00_1_SIM is RUNNING
Job a1wj_19850101_fc02_1_SIM is RUNNING
Job a1wj_19910101_fc03_1_SIM is RUNNING
Job a1wj_19860101_fc04_1_SIM is RUNNING
Job a1wj_19870101_fc00_1_SIM is RUNNING
Job a1wj_19850101_fc01_1_SIM is RUNNING
Job a1wj_19900101_fc04_1_SIM is RUNNING
^CAutosubmit will interrupt at the next safe occasion
Job a1wj_19910101_fc00_1_SIM is RUNNING
Job a1wj_19920101_fc00_1_SIM is RUNNING
Job a1wj_19900101_fc00_1_SIM is RUNNING
Job a1wj_19900101_fc01_1_SIM is RUNNING
Job a1wj_19900101_fc02_1_SIM is RUNNING
Job a1wj_19890101_fc04_1_SIM is RUNNING
Job a1wj_19890101_fc01_1_SIM is RUNNING
Job a1wj_19850101_fc00_1_SIM is RUNNING
Job a1wj_19890101_fc03_1_SIM is RUNNING
Job a1wj_19890101_fc02_1_SIM is RUNNING
etourign@bscesautosubmit02:/esarchive/autosubmit/a1wj/conf> autosubmit setstatus -fl a1wj_19910101_fc01_1_SIM -t READY -s -np a1wj
Preparing .lock file to avoid multiple instances with same expid.
Checking configuration files...
platforms_a1wj.conf OK
jobs_a1wj.conf OK
autosubmit_a1wj.conf OK
expdef_a1wj.conf OK
Configuration files OK
Creating jobs...
Loading JobList
Adding dependencies...
Removing redundant dependencies...
CHANGED: job: a1wj_19910101_fc01_1_SIM status to: READY
etourign@bscesautosubmit02:/esarchive/autosubmit/a1wj/conf> autosubmit run a1wj
Preparing .lock file to avoid multiple instances with same experiment id
Checking configuration files...
platforms_a1wj.conf OK
jobs_a1wj.conf OK
autosubmit_a1wj.conf OK
expdef_a1wj.conf OK
Configuration files OK
Starting job submission...
Creating jobs...
Loading JobList
Adding dependencies...
Removing redundant dependencies...
Checking scripts...
[WARNING] The following set of variables to be substituted in template script is not part of parameters set, and will be replaced by a blank value: set(['PISCES_timestep', 'LIM_timestep', 'OSM_NUMPROC', 'IFS_timestep', 'OSM_THREADS', 'NEM_FORCING_SET', 'RUN_coupFreq', 'NEMO_timestep', 'NEM_NUMPROC', 'XIO_NUMPROC'])
[WARNING] The following set of variables to be substituted in template script is not part of parameters set, and will be replaced by a blank value: set(['USE_DT_MACHINE', 'UPDATE_MODEL', 'DT_HOST', 'DTHOSTMACHINE', 'DT_USER'])
[WARNING] Scripts check failed
Running after failed scripts check is at your own risk!
644 of 723 jobs remaining (10:56)
[WARNING] Command sacct -n -X -j 5919477,5919489,5919488,5919483,5919482,5919481,5919480,5919487,5919486,5919478,5919479,5919485,5919484, -o 'jobid,State' in mn1.bsc.es warning: /home/cns54/cns54230/.bashrc: line 53: ppwd: command not found
Job a1wj_19790101_fc04_INI is COMPLETED
Job a1wj_19910101_fc04_INI is COMPLETED
Job a1wj_19900101_fc04_INI is COMPLETED
Job a1wj_19850101_fc04_INI is COMPLETED
Job a1wj_19840101_fc04_INI is COMPLETED
Job a1wj_19830101_fc04_INI is COMPLETED
Job a1wj_19820101_fc04_INI is COMPLETED
Job a1wj_19890101_fc04_INI is COMPLETED
Job a1wj_19880101_fc04_INI is COMPLETED
Job a1wj_19800101_fc04_INI is COMPLETED
Job a1wj_19810101_fc04_INI is COMPLETED
Job a1wj_19870101_fc04_INI is COMPLETED
Job a1wj_19860101_fc04_INI is COMPLETED
Jobs ready for marenostrum4: 91
[WARNING] Command /gpfs/scratch/cns54/cns54230/a1wj/LOG_a1wj/submit_marenostrum4.sh in mn1.bsc.es warning: /home/cns54/cns54230/.bashrc: line 53: ppwd: command not found
a1wj_19790101_fc00_1_SIM submitted
a1wj_19790101_fc01_1_SIM submitted
a1wj_19790101_fc02_1_SIM submitted
a1wj_19790101_fc03_1_SIM submitted
a1wj_19790101_fc04_1_SIM submitted
a1wj_19800101_fc00_1_SIM submitted
a1wj_19800101_fc01_1_SIM submitted
a1wj_19800101_fc02_1_SIM submitted
a1wj_19800101_fc03_1_SIM submitted
a1wj_19800101_fc04_1_SIM submitted
a1wj_19810101_fc00_1_SIM submitted
a1wj_19810101_fc01_1_SIM submitted
a1wj_19810101_fc02_1_SIM submitted
a1wj_19810101_fc03_1_SIM submitted
a1wj_19810101_fc04_1_SIM submitted
a1wj_19820101_fc00_1_SIM submitted
a1wj_19820101_fc01_1_SIM submitted
a1wj_19820101_fc02_1_SIM submitted
a1wj_19820101_fc03_1_SIM submitted
a1wj_19820101_fc04_1_SIM submitted
a1wj_19830101_fc00_1_SIM submitted
a1wj_19830101_fc01_1_SIM submitted
a1wj_19830101_fc02_1_SIM submitted
a1wj_19830101_fc03_1_SIM submitted
a1wj_19830101_fc04_1_SIM submitted
a1wj_19840101_fc00_1_SIM submitted
a1wj_19840101_fc01_1_SIM submitted
a1wj_19840101_fc02_1_SIM submitted
a1wj_19840101_fc03_1_SIM submitted
a1wj_19840101_fc04_1_SIM submitted
a1wj_19850101_fc00_1_SIM submitted
a1wj_19850101_fc01_1_SIM submitted
a1wj_19850101_fc02_1_SIM submitted
a1wj_19850101_fc03_1_SIM submitted
a1wj_19850101_fc04_1_SIM submitted
a1wj_19860101_fc00_1_SIM submitted
a1wj_19860101_fc01_1_SIM submitted
a1wj_19860101_fc02_1_SIM submitted
a1wj_19860101_fc03_1_SIM submitted
a1wj_19860101_fc04_1_SIM submitted
a1wj_19870101_fc00_1_SIM submitted
a1wj_19870101_fc01_1_SIM submitted
a1wj_19870101_fc02_1_SIM submitted
a1wj_19870101_fc03_1_SIM submitted
a1wj_19870101_fc04_1_SIM submitted
a1wj_19880101_fc00_1_SIM submitted
a1wj_19880101_fc01_1_SIM submitted
a1wj_19880101_fc02_1_SIM submitted
a1wj_19880101_fc03_1_SIM submitted
a1wj_19880101_fc04_1_SIM submitted
a1wj_19890101_fc00_1_SIM submitted
a1wj_19890101_fc01_1_SIM submitted
a1wj_19890101_fc02_1_SIM submitted
a1wj_19890101_fc03_1_SIM submitted
a1wj_19890101_fc04_1_SIM submitted
a1wj_19900101_fc00_1_SIM submitted
a1wj_19900101_fc01_1_SIM submitted
a1wj_19900101_fc02_1_SIM submitted
a1wj_19900101_fc03_1_SIM submitted
a1wj_19900101_fc04_1_SIM submitted
631 of 723 jobs remaining (10:57)
[WARNING] Command sacct -n -X -j 5919648,5919645,5919644,5919647,5919646,5919641,5919640,5919643,5919642,5919649,5919628,5919627,5919626,5919625,5919624,5919623,5919622,5919621,5919620,5919609,5919605,5919661,5919660,5919629,5919604,5919662,5919607,5919606,5919633,5919663,5919608,5919632,5919652,5919653,5919650,5919651,5919656,5919657,5919654,5919655,5919638,5919658,5919659,5919639,5919634,5919635,5919636,5919637,5919630,5919631,5919618,5919619,5919616,5919617,5919614,5919615,5919612,5919613,5919610,5919611, -o 'jobid,State' in mn1.bsc.es warning: /home/cns54/cns54230/.bashrc: line 53: ppwd: command not found
Job a1wj_19870101_fc04_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19870101_fc04_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19870101_fc01_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19870101_fc01_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19870101_fc00_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19870101_fc00_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19870101_fc03_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19870101_fc03_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19870101_fc02_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19870101_fc02_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19860101_fc02_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19860101_fc02_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19860101_fc01_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19860101_fc01_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19860101_fc04_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19860101_fc04_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19860101_fc03_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19860101_fc03_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19880101_fc00_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19880101_fc00_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19830101_fc04_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19830101_fc04_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19830101_fc03_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19830101_fc03_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19830101_fc02_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19830101_fc02_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19830101_fc01_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19830101_fc01_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19830101_fc00_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19830101_fc00_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19820101_fc04_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19820101_fc04_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19820101_fc03_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19820101_fc03_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19820101_fc02_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19820101_fc02_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19820101_fc01_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19820101_fc01_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19800101_fc00_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19800101_fc00_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19790101_fc01_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19790101_fc01_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19900101_fc02_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19900101_fc02_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19900101_fc01_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19900101_fc01_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19840101_fc00_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19840101_fc00_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19790101_fc00_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19790101_fc00_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19900101_fc03_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19900101_fc03_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19790101_fc03_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19790101_fc03_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19790101_fc02_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19790101_fc02_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19840101_fc04_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19840101_fc04_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19900101_fc04_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19900101_fc04_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19790101_fc04_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19790101_fc04_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19840101_fc03_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19840101_fc03_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19880101_fc03_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19880101_fc03_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19880101_fc04_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19880101_fc04_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19880101_fc01_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19880101_fc01_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19880101_fc02_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19880101_fc02_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19890101_fc02_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19890101_fc02_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19890101_fc03_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19890101_fc03_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19890101_fc00_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19890101_fc00_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19890101_fc01_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19890101_fc01_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19850101_fc04_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19850101_fc04_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19890101_fc04_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19890101_fc04_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19900101_fc00_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19900101_fc00_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19860101_fc00_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19860101_fc00_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19850101_fc00_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19850101_fc00_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19850101_fc01_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19850101_fc01_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19850101_fc02_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19850101_fc02_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19850101_fc03_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19850101_fc03_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19840101_fc01_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19840101_fc01_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19840101_fc02_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19840101_fc02_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19810101_fc04_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19810101_fc04_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19820101_fc00_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19820101_fc00_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19810101_fc02_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19810101_fc02_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19810101_fc03_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19810101_fc03_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19810101_fc00_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19810101_fc00_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19810101_fc01_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19810101_fc01_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19800101_fc03_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19800101_fc03_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19800101_fc04_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19800101_fc04_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19800101_fc01_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19800101_fc01_1_SIM completion check failed. There is no COMPLETED file
Job a1wj_19800101_fc02_1_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a1wj_19800101_fc02_1_SIM completion check failed. There is no COMPLETED file