GA Experiments failing: Non-wrapper related
I've been running some experiments with the Genetic Algorithm with the wrappers turned off, and they've been failing for different reasons. It's strange because I ran the same settings over the weekend and had no problem. It seems that one of two things can happen:
- It runs for a handful of generations and then stops giving the following error on the log files: 20201118_124702_run.log: (from a27f)
2020-11-19 01:56:08,727 Traceback (most recent call last):
File "/shared/earth/software/autosubmit/3.13.0b-foss-2015a-Python-2.7.9/lib/python2.7/site-packages/autosubmit-3.13.0b0-py2.7.egg/autosubmit/database/db_jobdata.py", line 1593, in _deactivate_current_last
cur.execute(sql, tuplerow)
OperationalError: unable to open database file
[WARNING] 2020-11-19 01:56:08,727 Error on Insert : OperationalError
2020-11-19 01:56:08,728 Wrapper table not found, trying packages. JobDataStructure.retrieve_packages
2020-11-19 01:56:08,731 Traceback (most recent call last):
File "/shared/earth/software/autosubmit/3.13.0b-foss-2015a-Python-2.7.9/lib/python2.7/site-packages/autosubmit-3.13.0b0-py2.7.egg/autosubmit/database/db_jobdata.py", line 1743, in _insert_job_data
cur.execute(sql, tuplerow)
OperationalError: unable to open database file
[WARNING] 2020-11-19 01:56:08,731 Error on Insert : OperationalError a27f_19600101_fc0000_23_SIM 23
2020-11-19 01:56:09,520 /gpfs/scratch/bsc32/bsc32786/a27f/LOG_a27f/a27f_19600101_fc0001_18_SIM.cmd.115.err File still no exists.. waiting 10s for a new retry ( retries left: 2)
2020-11-19 01:56:09,560 /gpfs/scratch/bsc32/bsc32786/a27f/LOG_a27f/a27f_19600101_fc0001_19_SIM.cmd.118.err File still no exists.. waiting 5s for a new retry ( retries left: 3)
2020-11-19 01:56:09,865 a27f_19600101_fc0000_23_SIM job seems to have completed: checking...
2020-11-19 01:56:09,876 Job a27f_19600101_fc0000_23_SIM is COMPLETED
2020-11-19 01:56:09,925 a27f_19600101_fc0000_23_SIM_STAT file have been transfered
2020-11-19 01:56:09,926 Reading config from /etc/autosubmitrc
2020-11-19 01:56:09,938 Reading config from /etc/autosubmitrc
2020-11-19 01:56:09,960 Reading config from /etc/autosubmitrc
2020-11-19 01:56:09,973 Reading config from /etc/autosubmitrc
2020-11-19 01:56:09,975 Job a27f_19600101_fc0000_23_SIM finished at 2020-11-19 00:15:44
2020-11-19 01:56:09,976 Job a27f_19600101_fc0000_24_SIM started at 2020-11-19 00:08:26
2020-11-19 01:56:09,976 Job a27f_19600101_fc0000_24_SIM is RUNNING
2020-11-19 01:56:09,993 a27f_19600101_fc0000_24_SIM_STAT file have been transfered
2020-11-19 01:56:09,994 Custom directives from platform.conf: None
2020-11-19 01:56:09,997 Reading config from /etc/autosubmitrc
2020-11-19 01:56:09,998 Database schema needs update.
[ERROR] 2020-11-19 01:56:10,002 Trace: tuple index out of range
[CRITICAL] 2020-11-19 01:56:10,002 Unhandled error: If you see this message, please report it in Autosubmit's GitLab project
20201118_124702_run_err.log:
[ERROR] 2020-11-19 01:53:17,597 Trace:
[ERROR] 2020-11-19 01:53:17,615 Command find /gpfs/scratch/bsc32/bsc32786/a27f/LOG_a27f -name *a27f_19600101_fc0001_52_SIM_COMPLETED in mn-bsc32 warning: [eCode=6005]
[ERROR] 2020-11-19 01:56:10,002 Trace: tuple index out of range
[CRITICAL] 2020-11-19 01:56:10,002 Unhandled error: If you see this message, please report it in Autosubmit's GitLab project
- It only runs INI and then stops: (from a27m)
20201126_095248_run.log: (the same in run_err.log)
2020-11-26 10:00:10,991 a27m_19600101_fc0000_100_SIM_STAT have been removed
2020-11-26 10:00:10,996 a27m_19600101_fc0000_100_SIM_COMPLETED been removed
[ERROR] 2020-11-26 10:00:33,175 Trace: ["sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n", "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)\n"]
[CRITICAL] 2020-11-26 10:00:33,179 Submission failed, check job and queue specified of job_sections of SIM [eCode=7014]
2020-11-26 10:00:33,179 More info at https://autosubmit.readthedocs.io/en/latest/faq.html