Vertical wrapper with dependencies not working on ECMWF
Jobs ready for cca-intel: 2
a0o7_19500101_fc0_1_BACKUP submitted
a0o7_19500101_fc0_2_SIM submitted
a0o7_19500101_fc0_3_SIM submitted
a0o7_19500101_fc0_4_SIM submitted
a0o7_19500101_fc0_5_SIM submitted
a0o7_19500101_fc0_6_SIM submitted
a0o7_19500101_fc0_7_SIM submitted
a0o7_19500101_fc0_8_SIM submitted
a0o7_19500101_fc0_9_SIM submitted
a0o7_19500101_fc0_10_SIM submitted
a0o7_19500101_fc0_11_SIM submitted
a0o7_19500101_fc0_12_SIM submitted
a0o7_19500101_fc0_13_SIM submitted
a0o7_19500101_fc0_14_SIM submitted
a0o7_19500101_fc0_15_SIM submitted
a0o7_19500101_fc0_16_SIM submitted
a0o7_19500101_fc0_17_SIM submitted
a0o7_19500101_fc0_18_SIM submitted
a0o7_19500101_fc0_19_SIM submitted
a0o7_19500101_fc0_20_SIM submitted
a0o7_19500101_fc0_21_SIM submitted
a0o7_19500101_fc0_22_SIM submitted
a0o7_19500101_fc0_23_SIM submitted
a0o7_19500101_fc0_24_SIM submitted
a0o7_19500101_fc0_25_SIM submitted
a0o7_19500101_fc0_26_SIM submitted
a0o7_19500101_fc0_27_SIM submitted
a0o7_19500101_fc0_28_SIM submitted
a0o7_19500101_fc0_29_SIM submitted
a0o7_19500101_fc0_30_SIM submitted
a0o7_19500101_fc0_31_SIM submitted
a0o7_19500101_fc0_32_SIM submitted
a0o7_19500101_fc0_33_SIM submitted
a0o7_19500101_fc0_34_SIM submitted
a0o7_19500101_fc0_35_SIM submitted
a0o7_19500101_fc0_36_SIM submitted
a0o7_19500101_fc0_37_SIM submitted
a0o7_19500101_fc0_38_SIM submitted
a0o7_19500101_fc0_39_SIM submitted
a0o7_19500101_fc0_40_SIM submitted
a0o7_19500101_fc0_41_SIM submitted
a0o7_19500101_fc0_42_SIM submitted
a0o7_19500101_fc0_43_SIM submitted
a0o7_19500101_fc0_44_SIM submitted
a0o7_19500101_fc0_45_SIM submitted
a0o7_19500101_fc0_46_SIM submitted
a0o7_19500101_fc0_47_SIM submitted
a0o7_19500101_fc0_48_SIM submitted
a0o7_19500101_fc0_49_SIM submitted
a0o7_19500101_fc0_50_SIM submitted
99 of 103 jobs remaining (12:43)
Job a0o7_19500101_fc0_2_SIM is COMPLETED
Job a0o7_19500101_fc0_3_SIM is COMPLETED
Job a0o7_19500101_fc0_4_SIM is COMPLETED
Job a0o7_19500101_fc0_5_SIM is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a0o7_19500101_fc0_5_SIM completion check failed. There is no COMPLETED file
[WARNING] Could not get start time for a0o7_19500101_fc0_5_SIM. Using current time as an approximation
Job a0o7_19500101_fc0_1_BACKUP is FAILED. Checking completed files to confirm the failure...
[WARNING] Job a0o7_19500101_fc0_1_BACKUP seems to have failed but there is a COMPLETED file
how ecaccess-job-list looks like:
dmanuben@bscearth309:/esnas/autosubmit/a0o7/tmp> ecaccess-job-list
8553683 cca STOP 1/1 Jul 19 10:46 (-) (ASThread_15004598347635_1908_1.cmd )
8553694 cca DONE 1/1 Jul 19 10:46 (-) (ASThread_1500461085746_1_1.cmd )
8553691 cca DONE 1/1 Jul 19 10:45 (-) (ASThread_15004610855440_1_1.cmd )
8553682 cca STOP 1/1 Jul 19 10:45 (-) (ASThread_15004598342889_1908_3.cmd )
8553690 cca DONE 1/1 Jul 19 10:45 (-) (ASThread_15004610858180_1_1.cmd )
8553679 cca STOP 1/1 Jul 19 10:44 (-) (ASThread_15004598345089_1908_3.cmd )
8553676 cca STOP 1/1 Jul 19 10:43 (-) (ASThread_15004598343515_1908_3.cmd )
8553673 cca STOP 1/1 Jul 19 10:42 (-) (ASThread_15004598347617_1908_3.cmd )
8553670 cca STOP 1/1 Jul 19 10:41 (-) (ASThread_1500459834423_1908_3.cmd )
8553663 cca STOP 1/1 Jul 19 10:40 (-) (ASThread_15004598347214_1908_3.cmd )
8553660 cca STOP 1/1 Jul 19 10:39 (-) (ASThread_15004598341472_1908_3.cmd )
8553651 cca STOP 1/1 Jul 19 10:37 (-) (ASThread_15004598347447_1908_3.cmd )
8553648 cca STOP 1/1 Jul 19 10:36 (-) (ASThread_15004598347703_1908_3.cmd )
8553645 cca STOP 1/1 Jul 19 10:35 (-) (ASThread_1500459834842_1908_3.cmd )
8553641 cca STOP 1/1 Jul 19 10:34 (-) (ASThread_1500459834172_1908_3.cmd )
8553638 cca STOP 1/1 Jul 19 10:33 (-) (ASThread_15004598349145_1908_3.cmd )
8553634 cca STOP 1/1 Jul 19 10:32 (-) (ASThread_15004598341716_1908_3.cmd )
8553631 cca STOP 1/1 Jul 19 10:31 (-) (ASThread_15004598347969_1908_3.cmd )
8553622 cca STOP 1/1 Jul 19 10:30 (-) (ASThread_1500459834451_1908_3.cmd )
8553619 cca DONE 1/1 Jul 19 10:25 (-) (ASThread_15004598343673_1908_3.cmd )
8553612 cca STOP 1/1 Jul 19 10:24 (-) (ASThread_15004598346533_1_1.cmd )
8553595 cca DONE 1/1 Jul 19 10:19 (-) (a0o7_19500101_fc0_1_SIM.cmd )
8553590 cca DONE 1/1 Jul 19 10:17 (-) (a0o7_19500101_fc0_INI.cmd )
8553587 cca DONE 1/1 Jul 19 10:16 (-) (a0o7_REMOTE_SETUP.cmd )
Logs:
`c3d@cca-login3:/scratch/ms/spesiccf/c3d/a0o7/LOG_a0o7> ls -ltrh
total 385K
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:15 a0o7_REMOTE_SETUP.cmd
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:16 a0o7_REMOTE_SETUP.20170719121547.err
-rw-r----- 1 c3d spesiccf 22 Jul 19 10:16 a0o7_REMOTE_SETUP_STAT
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:16 a0o7_REMOTE_SETUP_COMPLETED
-rw-r----- 1 c3d spesiccf 8.5K Jul 19 10:16 a0o7_REMOTE_SETUP.20170719121547.out
-rw-r----- 1 c3d spesiccf 966 Jul 19 10:17 a0o7_19500101_fc0_INI.cmd
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:17 a0o7_19500101_fc0_INI.20170719121718.err
-rw-r----- 1 c3d spesiccf 22 Jul 19 10:18 a0o7_19500101_fc0_INI_STAT
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:18 a0o7_19500101_fc0_INI_COMPLETED
-rw-r----- 1 c3d spesiccf 8.2K Jul 19 10:18 a0o7_19500101_fc0_INI.20170719121718.out
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:18 a0o7_19500101_fc0_1_SIM.cmd
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:23 a0o7_19500101_fc0_1_SIM.20170719121840.err
-rw-r----- 1 c3d spesiccf 22 Jul 19 10:23 a0o7_19500101_fc0_1_SIM_STAT
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:23 a0o7_19500101_fc0_1_SIM_COMPLETED
-rw-r----- 1 c3d spesiccf 8.6K Jul 19 10:23 a0o7_19500101_fc0_1_SIM.20170719121840.out
-rw-r----- 1 c3d spesiccf 989 Jul 19 10:24 a0o7_19500101_fc0_1_BACKUP.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:24 a0o7_19500101_fc0_2_SIM.cmd
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:24 a0o7_19500101_fc0_1_BACKUP.cmd.0.out
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:24 a0o7_19500101_fc0_1_BACKUP.cmd.0.err
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:24 a0o7_19500101_fc0_3_SIM.cmd
-rw-r----- 1 c3d spesiccf 22 Jul 19 10:24 a0o7_19500101_fc0_1_BACKUP_STAT
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:24 a0o7_19500101_fc0_1_BACKUP_COMPLETED
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:24 a0o7_19500101_fc0_4_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:25 ASThread_15004598343673_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:25 a0o7_19500101_fc0_5_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:25 a0o7_19500101_fc0_6_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:26 a0o7_19500101_fc0_7_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:26 ASThread_1500459834451_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:26 a0o7_19500101_fc0_8_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:27 a0o7_19500101_fc0_9_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:27 a0o7_19500101_fc0_10_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:27 ASThread_15004598347969_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:28 a0o7_19500101_fc0_11_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:28 a0o7_19500101_fc0_12_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:28 a0o7_19500101_fc0_13_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:28 ASThread_15004598341716_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:29 a0o7_19500101_fc0_14_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:29 a0o7_19500101_fc0_15_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:29 a0o7_19500101_fc0_16_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:29 ASThread_15004598349145_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:30 a0o7_19500101_fc0_17_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:30 a0o7_19500101_fc0_18_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:30 a0o7_19500101_fc0_19_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:31 ASThread_1500459834172_1908_3.cmd
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:31 a0o7_19500101_fc0_2_SIM.cmd.0.out
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:31 a0o7_19500101_fc0_2_SIM.cmd.0.err
-rw-r----- 1 c3d spesiccf 22 Jul 19 10:31 a0o7_19500101_fc0_2_SIM_STAT
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:31 a0o7_19500101_fc0_3_SIM.cmd.1.out
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:31 a0o7_19500101_fc0_3_SIM.cmd.1.err
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:31 a0o7_19500101_fc0_2_SIM_COMPLETED
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:31 a0o7_19500101_fc0_4_SIM.cmd.2.out
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:31 a0o7_19500101_fc0_4_SIM.cmd.2.err
-rw-r----- 1 c3d spesiccf 22 Jul 19 10:31 a0o7_19500101_fc0_3_SIM_STAT
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:31 a0o7_19500101_fc0_3_SIM_COMPLETED
-rw-r----- 1 c3d spesiccf 22 Jul 19 10:31 a0o7_19500101_fc0_4_SIM_STAT
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:31 a0o7_19500101_fc0_4_SIM_COMPLETED
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:31 a0o7_19500101_fc0_20_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:31 a0o7_19500101_fc0_21_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:32 a0o7_19500101_fc0_22_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:32 ASThread_1500459834842_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:32 a0o7_19500101_fc0_23_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:32 a0o7_19500101_fc0_24_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:33 a0o7_19500101_fc0_25_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:33 ASThread_15004598347703_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:33 a0o7_19500101_fc0_26_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:34 a0o7_19500101_fc0_27_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:34 a0o7_19500101_fc0_28_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:34 ASThread_15004598347447_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:35 a0o7_19500101_fc0_29_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:35 a0o7_19500101_fc0_30_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:35 a0o7_19500101_fc0_31_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:35 ASThread_15004598341472_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:36 a0o7_19500101_fc0_32_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:36 a0o7_19500101_fc0_33_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:36 a0o7_19500101_fc0_34_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:36 ASThread_15004598347214_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:37 a0o7_19500101_fc0_35_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:37 a0o7_19500101_fc0_36_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:37 a0o7_19500101_fc0_37_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:37 ASThread_1500459834423_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:38 a0o7_19500101_fc0_38_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:38 a0o7_19500101_fc0_39_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:38 a0o7_19500101_fc0_40_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:38 ASThread_15004598347617_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:39 a0o7_19500101_fc0_41_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:39 a0o7_19500101_fc0_42_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:39 a0o7_19500101_fc0_43_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:39 ASThread_15004598343515_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:40 a0o7_19500101_fc0_44_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:40 a0o7_19500101_fc0_45_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:40 a0o7_19500101_fc0_46_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:40 ASThread_15004598345089_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:41 a0o7_19500101_fc0_47_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:41 a0o7_19500101_fc0_48_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:41 a0o7_19500101_fc0_49_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:41 ASThread_15004598342889_1908_3.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:42 a0o7_19500101_fc0_50_SIM.cmd
-rw-r----- 1 c3d spesiccf 1.2K Jul 19 10:42 ASThread_15004598347635_1908_1.cmd
-rw-r----- 1 c3d spesiccf 989 Jul 19 10:44 a0o7_19500101_fc0_2_BACKUP.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:45 ASThread_15004610858180_1_1.cmd
-rw-r----- 1 c3d spesiccf 989 Jul 19 10:45 a0o7_19500101_fc0_3_BACKUP.cmd
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:45 a0o7_19500101_fc0_2_BACKUP.cmd.0.out
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:45 a0o7_19500101_fc0_2_BACKUP.cmd.0.err
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:45 ASThread_15004610855440_1_1.cmd
-rw-r----- 1 c3d spesiccf 22 Jul 19 10:45 a0o7_19500101_fc0_2_BACKUP_STAT
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:45 a0o7_19500101_fc0_2_BACKUP_COMPLETED
-rw-r----- 1 c3d spesiccf 989 Jul 19 10:45 a0o7_19500101_fc0_4_BACKUP.cmd
-rw-r----- 1 c3d spesiccf 1.1K Jul 19 10:46 ASThread_1500461085746_1_1.cmd
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:46 a0o7_19500101_fc0_3_BACKUP.cmd.0.out
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:46 a0o7_19500101_fc0_3_BACKUP.cmd.0.err
-rw-r----- 1 c3d spesiccf 22 Jul 19 10:46 a0o7_19500101_fc0_3_BACKUP_STAT
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:46 a0o7_19500101_fc0_3_BACKUP_COMPLETED
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:46 a0o7_19500101_fc0_4_BACKUP.cmd.0.out
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:46 a0o7_19500101_fc0_4_BACKUP.cmd.0.err
-rw-r----- 1 c3d spesiccf 22 Jul 19 10:46 a0o7_19500101_fc0_4_BACKUP_STAT
-rw-r----- 1 c3d spesiccf 0 Jul 19 10:46 a0o7_19500101_fc0_4_BACKUP_COMPLETED``
The header of a wrap:
#PBS -N ASThread_1500461085746_1_1
#PBS -q ns
#PBS -l EC_billing_account=spesiccf
#PBS -o $SCRATCH/a0o7/LOG_a0o7/ASThread_1500461085746_1_1.out
#PBS -o $SCRATCH/a0o7/LOG_a0o7/ASThread_1500461085746_1_1.err
#PBS -l walltime=15:00:00
#PBS -l EC_total_tasks=1
#PBS -l EC_hyperthreads=1
A couple of issues I see:
-
-o
is used for both err and out - $SCRATCH needs to be expanded
My guess for he reason jobs are going to STOP
is that $SCRATCH variable is not accessible from the PBS header and cannot be expanded.
I am looking into it...
@etourign @pechevar @mcastril @otinto