Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • autosubmit autosubmit
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 338
    • Issues 338
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 21
    • Merge requests 21
  • Deployments
    • Deployments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • Earth SciencesEarth Sciences
  • autosubmitautosubmit
  • Issues
  • #434
Closed
Open
Issue created Sep 30, 2019 by wuruchi@wuruchiMaintainer

Analysis of Grouping function.

Hello team,

@mcastril @dbeltran

I open this issue to start with the detailed analysis of the grouping function of autosubmit. I will post code snippets, questions, request information, etc.; and then we will try to figure out what's really happening so it also helps as documentation.

Let's start:

@dbeltran, as I have been testing the wrappers, I noticed that when I execute "CREATE" the pdf plot does not show the wrapper boxes, and the corresponding database in the pkl folder is created but empty. I wonder if the wrappers are actually created when "RUN" is executed.

Also, in the monitor method of autosubmit.py I found this code snippet:

Line 1382 of autosubmit.py

`

    if as_conf.get_wrapper_type() != 'none' and check_wrapper:
        packages_persistence = JobPackagePersistence(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
                                                     "job_packages_" + expid)
        os.chmod(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl", "job_packages_" + expid + ".db"), 0664)
        packages_persistence.reset_table(True)
        referenced_jobs_to_remove = set()
        job_list_wrappers = copy.deepcopy(job_list)
        jobs_wr = copy.deepcopy(jobs)
        [job for job in jobs_wr if (job.status != Status.COMPLETED)]

        for job in jobs_wr:
            for child in job.children:
                if child not in jobs_wr:
                    referenced_jobs_to_remove.add(child)
            for parent in job.parents:
                if parent not in jobs_wr:
                    referenced_jobs_to_remove.add(parent)
        for job in jobs_wr:
            job.children = job.children - referenced_jobs_to_remove
            job.parents = job.parents - referenced_jobs_to_remove
        Autosubmit.generate_scripts_andor_wrappers(as_conf, job_list_wrappers, jobs_wr,
                                                   packages_persistence, True)

        packages = packages_persistence.load(True)
    else:
        packages = JobPackagePersistence(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
                                         "job_packages_" + expid).load()`

Is -cw used in conjunction with "monitor" frequently, this code seems to reset the package database and recreate the wrapper list, and even send it to the queue at some point. and the line "job for job in jobs_wr if (job.status != Status.COMPLETED)]" does not seem to serve any purpose.

Edited Sep 30, 2019 by wuruchi
Assignee
Assign to
Time tracking