Default job yaml combined with CUSTOME_CONFIG project job.yml
Hello @dbeltran and @bdepaula,
Autosubmit Version
tag:v4.0.0b2 autosubmitconfigparser: 0.0.75
Summary
When trying to run an experiment with a local project, where in the <path2experiments>/<expid>/conf/expdef_<expid>.yml I have CUSTOM_CONFIG: "%ROOTDIR%/proj/proj_ifs-fesom/conf/ifs-fesom", project.PROJECT_TYPE: local, project.PROJECT_DESTINATION: 'proj_ifs-fesom' local.PROJECT_PATH: '~/utilities/workflow', Autosubmit combines the content of <path2experiments>/<expid>/conf/jobs_<expid>.yml, with %ROOTDIR%/proj/proj_ifs-fesom/conf/ifs-fesom/jobs.yml. Because our project lacks of the template scripts for some of the tasks in the default (POST, CLEAN and TRANSFER) we get the following error:
Autosubmit is running with 4.0.0b
Checking configuration files...
expdef_a01t.yml OK
platforms_a01t.yml OK
autosubmit_a01t.yml OK
Traceback (most recent call last):
File "/home/ollie/mandresm/.conda/envs/autosubmit/lib/python3.7/site-packages/autosubmitconfigparser/config/configcommon.py", line 901, in check_conf_files
result = self.show_messages()
File "/home/ollie/mandresm/.conda/envs/autosubmit/lib/python3.7/site-packages/autosubmitconfigparser/config/configcommon.py", line 549, in show_messages
raise AutosubmitCritical(message, 7014)
log.log.AutosubmitCritical:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ollie/mandresm/utilities/autosubmit/bin/autosubmit", line 36, in main
Autosubmit.parse_args()
File "/home/ollie/mandresm/utilities/autosubmit/autosubmit/autosubmit.py", line 630, in parse_args
args.start_after, args.run_members)
File "/home/ollie/mandresm/utilities/autosubmit/autosubmit/autosubmit.py", line 1676, in run_experiment
as_conf.check_conf_files(True, True)
File "/home/ollie/mandresm/.conda/envs/autosubmit/lib/python3.7/site-packages/autosubmitconfigparser/config/configcommon.py", line 906, in check_conf_files
raise AutosubmitCritical(e.message, e.code, e.trace)
log.log.AutosubmitCritical:
[CRITICAL] On Configuration files:
Critical Issues on [Jobs] config file:
[POST] FILE POST.sh doesn't exist and check parameter is not set on_submission value
[CLEAN] FILE CLEAN.sh doesn't exist and check parameter is not set on_submission value
[TRANSFER] FILE TRANSFER.sh doesn't exist and check parameter is not set on_submission value
[eCode=7014]Steps to reproduce
git clone https://earth.bsc.es/gitlab/digital-twins/de_340/workflow -b prototype/ifs-fesom-customconf ~/DT_workflow- If you are using tag
v4.0.0b2fix this line https://earth.bsc.es/gitlab/es/autosubmit/-/blob/v4.0.0b2/autosubmit/job/job_dict.py#L392 to look like this one https://earth.bsc.es/gitlab/es/autosubmit/-/blob/master/autosubmit/job/job_dict.py#L392 autosubmit expid -H LUMI -d "Basic ifs-fesom/RAPS workflow"- Set The
expdef.ymlto look like this:
DeFault:
EXPID: a002
HPCARCH: LUMI # default hpc when no platform is given for a job(?)
CUSTOM_CONFIG: "%ROOTDIR%/proj/proj_ifs-fesom/conf/ifs-fesom,/albedo/home/sebeye001/.config/autosubmit/platforms.yml"
experiment:
DATELIST: 20200120
MEMBERS: "fc0"
CHUNKSIZEUNIT: day
CHUNKSIZE: 1
NUMCHUNKS: 5
CHUNKINI: ''
CALENDAR: standard
project:
PROJECT_TYPE: local
PROJECT_DESTINATION: 'proj_ifs-fesom'
local:
PROJECT_PATH: '~/DT_workflow'
project_files:
FILE_PROJECT_CONF: ''
FILE_JOBS_CONF: ''
JOB_SCRIPTS_TYPE: ''
rerun:
RERUN: FALSE
RERUN_JOBLIST: ''autosubmit create <expid>autosubmit run <expid>
The error should occur at this point.
If you enter in debug mode and put a break point in autosubmitconfigparser/config/configcommon.py:1022 and have a look at the content of parser['JOBS'] you'll see that both job.yml files are combined, the ones defined by the project and the default ones provided by Autosubmit.
'JOBS': {'CLEAN': {'ADDITIONAL_FILES': [],
'DEPENDENCIES': {'POST': {}},
'FILE': 'CLEAN.sh',
'RUNNING': 'once',
'WALLCLOCK': '00:05'},
'INI': {'ADDITIONAL_FILES': [],
'DEPENDENCIES': {'REMOTE_SETUP': {}},
'FILE': 'templates/ini.sh',
'PLATFORM': 'LUMI-login',
'RUNNING': 'member',
'WALLCLOCK': '00:05'},
'LOCAL_SETUP': {'ADDITIONAL_FILES': [],
'DEPENDENCIES': {},
'FILE': 'templates/local_setup.sh',
'PLATFORM': 'LOCAL',
'RUNNING': 'once'},
'POST': {'ADDITIONAL_FILES': [],
'DEPENDENCIES': {'SIM': {}},
'FILE': 'POST.sh',
'RUNNING': 'once',
'WALLCLOCK': '00:05'},
'REMOTE_SETUP': {'ADDITIONAL_FILES': [],
'DEPENDENCIES': {'LOCAL_SETUP': {},
'SYNCHRONIZE': {}},
'FILE': 'templates/remote_setup.sh',
'PLATFORM': 'LUMI-login',
'RUNNING': 'once',
'WALLCLOCK': '00:05'},
'SIM': {'ADDITIONAL_FILES': [],
'CUSTOM_DIRECTIVES': "['#SBATCH -p debug', '#SBATCH -N 1', "
"'#SBATCH --ntasks-per-node=64', "
"'#SBATCH --export=ALL']",
'DEPENDENCIES': {'INI': {}, 'SIM-1': {}},
'FILE': 'templates/sim.sh',
'PROCESSORS': 64,
'RUNNING': 'chunk',
'THREADS': 2,
'WALLCLOCK': '00:20'},
'SYNCHRONIZE': {'ADDITIONAL_FILES': [],
'DEPENDENCIES': {'LOCAL_SETUP': {}},
'FILE': 'templates/synchronize.sh',
'PLATFORM': 'LOCAL',
'RUNNING': 'once'},
'TRANSFER': {'ADDITIONAL_FILES': [],
'DEPENDENCIES': {'CLEAN': {}},
'FILE': 'TRANSFER.sh',
'PLATFORM': 'LOCAL',
'RUNNING': 'member'}},
What is the current bug behavior?
Default's jobs.yaml and project's jobs.yml information is combined.
What is the expected correct behavior?
I am not sure... I guess that if there is JOBS information in the project, that should not be combined with the default, after all, the JOBS dictionary is the workflow definition per se, and the workflow definition will depend on the project, right?
Thanks for your help!