Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • autosubmit autosubmit
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 338
    • Issues 338
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 21
    • Merge requests 21
  • Deployments
    • Deployments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • Earth SciencesEarth Sciences
  • autosubmitautosubmit
  • Issues
  • #939
Closed
Open
Issue created Feb 02, 2023 by Bruno de Paula Kinoshita@bdepaulaMaintainer

[enhancement] Create experiment, workflow, and tasks documentation from Autosubmit configuration files

This idea came from my induction training at the BSC, in a CES trello card. While trying Auto-Monarch and Auto-Earth, I noticed these tools had a similar documentation format, but parts of the documentation appeared to be out of sync with the current workflow code.

I reported the issues to both repositories. After the response from Auto-Monarch by @gmontane (thanks again!), I shared with him an idea to auto-generate the documentation in certain formats, like Markdown, directly from Autosubmit experiment configuration.

Other workflow managers support this already, like Cylc, ecFlow, and cwltool. The idea is to have new attributes in our YAML configuration for documentation. For example:

jobs.yaml

JOBS:
  LOCAL_SETUP:
    FILE: template/local_setup.sh
    RUNNING: once
    TITLE: A task to prepare the local archives used by the model
    DESCRIPTION: |
      This verifies if the expected files exist in the local directory,
      `tar`'s these files, and creates a log line about its execution.
      Executed once, this step is optional if you have a pre-existing
      model binary, and can be disabled with the environment variable
      `ABCDE`...
  SIM:
    FILE: template/ini.sh
    TITLE: The model simulation task
    DESCRIPTION: |
      This is the most important task of this workflow. It will ...

expdef.yaml

DEFAULT:
  EXPID: a000
  HPCARCH: local
  TITLE: Auto-Application
  DESCRIPTION: |
    Auto-Application is the Autosubmit workflow for the Application model.
    It is developed and maintained by the team ABC.

    It is a model of this type, that does this, and that...
    
    documentation:  <https://intranet...>
    issue tracking: <https://gitlab...>
    source code:    <https://gitlab...>
    contact point:  [ABC DEF](mailto:....)

In this example, the TITLE attribute would be a short title for the workflow node, by default the experiment ID or the task ID. The DESCRIPTION attribute would then be a longer text with links, documentation about the task or workflow, who to contact, what's the expected input, etc.

These values would then be used by, maybe, a new subcommand like autosubmit autodoc, to produce text, HTML, or Markdown output.

@gmontane also suggested to use the GitLab API to make it easier to update the Wiki when there's a change. I think that would definitely keep the sources and docs in sync!

Another suggestion from @gmontane: “and we should define a common format between projects.”

By getting some feedback from users for this common format, I think we could start simple with a Markdown template using Python stdlib's string module to render it. This way we wouldn't have to add a new dependency to Autosubmit, and we could improve the solution later if needed (but that's more about the implementation, that could be discussed later).

Edited Feb 02, 2023 by Bruno de Paula Kinoshita
Assignee
Assign to
Time tracking