[bug] Bug with CHECK: on_submission with horizontal wrappers
@dbeltran , just creating an issue as this was reported by @agayayav in the DestinE workflow repository.
Here's what I did to reproduce the bug with the master
branch, and affects latest version 4.1.2:
(autosubmit4) bdepaula@bsces107921:~/Development/python/workspace/autosubmit$ git log -n 1
commit 75464cc8728cf8f9b768fb47ff4d9e5f03663251 (HEAD -> master, upstream/master, upstream/HEAD)
Author: Daniel Beltran Mora <daniel.beltran@bsc.es>
Date: Mon Mar 11 15:01:27 2024 +0100
changelog
# File: minimal.yml
CONFIG:
AUTOSUBMIT_VERSION: "4.1.2"
MAXWAITINGJOBS: 20
TOTALJOBS: 20
SAFETYSLEEPTIME: 10
RETRIALS: 0
MAIL:
NOTIFICATIONS: False
TO:
STORAGE:
TYPE: pkl
COPY_REMOTE_LOGS: true
DEFAULT:
EXPID: "a005"
HPCARCH: "local"
EXPERIMENT:
DATELIST: '20000101'
MEMBERS: fc0
CHUNKSIZEUNIT: month
CHUNKSIZE: '4'
NUMCHUNKS: '2'
CHUNKINI: ''
CALENDAR: standard
PROJECT:
PROJECT_TYPE: none
PROJECT_DESTINATION: ''
GIT:
PROJECT_ORIGIN: ''
PROJECT_BRANCH: ''
PROJECT_COMMIT: ''
PROJECT_SUBMODULES: ''
FETCH_SINGLE_BRANCH: true
SVN:
PROJECT_URL: ''
PROJECT_REVISION: ''
LOCAL:
PROJECT_PATH: ''
PROJECT_FILES:
FILE_PROJECT_CONF: ''
FILE_JOBS_CONF: ''
JOB_SCRIPTS_TYPE: ''
RERUN:
RERUN: false
RERUN_JOBLIST: ''
PLATFORMS:
MARENOSTRUM4:
TYPE: slurm
HOST: mn1.bsc.es
PROJECT: bsc32
USER: bruno
QUEUE: debug
SCRATCH_DIR: /gpfs/scratch
ADD_PROJECT_TO_HOST: false
MAX_WALLCLOCK: 48:00
TEMP_DIR: ''
MAX_PROCESSORS: 128
JOBS:
TRANSFER:
PLATFORM: MARENOSTRUM4
PROCESSORS: 1
CHECK: on_submission
FILE: test.sh
WRAPPERS:
WRAPPER:
TYPE: horizontal
JOBS_IN_WRAPPER: TRANSFER
The key in the config above appear to be the CHECK: on_submission
, and the WRAPPER
.
To see the error message (hidden in the console by default, written to the log):
$ autosubmit create -cw a005 -np && autosubmit -lc DEBUG inspect a005
Checking configuration files...
Expdef config file is correct
Platforms sections: OK
Jobs sections OK
wrappers OK
Autosubmit general sections OK
Configuration files OK
The Experiment name is: a005
Sleep: 10
Loading JobList
Load finished
Creating jobs...
Creating TRANSFER jobs
Adding dependencies to the graph..
Changes detected, calculating dependencies
Adding dependencies for TRANSFER jobs
Adding dependencies to the job..
Transitive reduction...
Looking for edgeless jobs...
Loading JobList
Load finished
Creating jobs...
Creating TRANSFER jobs
Adding dependencies to the graph..
Changes detected, calculating dependencies
Adding dependencies for TRANSFER jobs
Adding dependencies to the job..
Transitive reduction...
Looking for edgeless jobs...
Length of the jobs list: 1
Sleep: 10
Starting to generate cmd scripts
Generating cmd scripts only for selected jobs
[WARNING] Generating the auxiliary job_list used for the -CW flag.
[WARNING] Aux Job_list was generated successfully
Loading parameters...
Parameters load.
Checking scripts...
1 of 1 checked
Scripts OK
Updating FAILED jobs
Updating WAITING jobs
Setting job: a005_TRANSFER status to: READY (all parents completed)...
Update finished
Loading parameters...
Parameters load.
Number of jobs available: 20
Number of jobs ready: 1
Jobs ready for MARENOSTRUM4: 1
Traceback (most recent call last):
File "/home/bdepaula/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 2382, in submit_ready_jobs
packages_to_submit = JobPackager(as_conf, platform, job_list, hold=hold).build_packages()
File "/home/bdepaula/Development/python/workspace/autosubmit/autosubmit/job/job_packager.py", line 468, in build_packages
built_packages_tmp = self._build_horizontal_packages(jobs, wrapper_limits, section,wrapper_info=current_info)
File "/home/bdepaula/Development/python/workspace/autosubmit/autosubmit/job/job_packager.py", line 536, in _build_horizontal_packages
package_jobs = horizontal_packager.build_horizontal_package()
File "/home/bdepaula/Development/python/workspace/autosubmit/autosubmit/job/job_packager.py", line 848, in build_horizontal_package
if job.total_processors == "":
File "/home/bdepaula/Development/python/workspace/autosubmit/autosubmit/job/job.py", line 782, in total_processors
return int(self.processors)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/bdepaula/Development/python/workspace/autosubmit/bin/autosubmit", line 57, in main
return_value = Autosubmit.parse_args()
File "/home/bdepaula/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 693, in parse_args
return Autosubmit.inspect(args.expid, args.list, args.filter_chunks, args.filter_status,
File "/home/bdepaula/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 1516, in inspect
Autosubmit.generate_scripts_andor_wrappers(
File "/home/bdepaula/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 1607, in generate_scripts_andor_wrappers
Autosubmit.submit_ready_jobs(as_conf, job_list, platforms_to_test, packages_persistence, True,
File "/home/bdepaula/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 2417, in submit_ready_jobs
raise AutosubmitCritical("This seems like a bug in the code, please contact AS developers", 7070, str(e))
log.log.AutosubmitCritical:
Trace: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
[CRITICAL] This seems like a bug in the code, please contact AS developers [eCode=7070]
More info at https://autosubmit.readthedocs.io/en/master/troubleshooting/error-codes.html
Will link with the existing merge request.