Analysis of Grouping function.
Hello team,
I open this issue to start with the detailed analysis of the grouping function of autosubmit. I will post code snippets, questions, request information, etc.; and then we will try to figure out what's really happening so it also helps as documentation.
Let's start:
@dbeltran, as I have been testing the wrappers, I noticed that when I execute "CREATE" the pdf plot does not show the wrapper boxes, and the corresponding database in the pkl folder is created but empty. I wonder if the wrappers are actually created when "RUN" is executed.
Also, in the monitor method of autosubmit.py I found this code snippet:
Line 1382 of autosubmit.py
`
if as_conf.get_wrapper_type() != 'none' and check_wrapper:
packages_persistence = JobPackagePersistence(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
"job_packages_" + expid)
os.chmod(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl", "job_packages_" + expid + ".db"), 0664)
packages_persistence.reset_table(True)
referenced_jobs_to_remove = set()
job_list_wrappers = copy.deepcopy(job_list)
jobs_wr = copy.deepcopy(jobs)
[job for job in jobs_wr if (job.status != Status.COMPLETED)]
for job in jobs_wr:
for child in job.children:
if child not in jobs_wr:
referenced_jobs_to_remove.add(child)
for parent in job.parents:
if parent not in jobs_wr:
referenced_jobs_to_remove.add(parent)
for job in jobs_wr:
job.children = job.children - referenced_jobs_to_remove
job.parents = job.parents - referenced_jobs_to_remove
Autosubmit.generate_scripts_andor_wrappers(as_conf, job_list_wrappers, jobs_wr,
packages_persistence, True)
packages = packages_persistence.load(True)
else:
packages = JobPackagePersistence(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
"job_packages_" + expid).load()`
Is -cw used in conjunction with "monitor" frequently, this code seems to reset the package database and recreate the wrapper list, and even send it to the queue at some point. and the line "job for job in jobs_wr if (job.status != Status.COMPLETED)]" does not seem to serve any purpose.