[enhancement] Create experiment, workflow, and tasks documentation from Autosubmit configuration files
This idea came from my induction training at the BSC, in a CES trello card. While trying Auto-Monarch and Auto-Earth, I noticed these tools had a similar documentation format, but parts of the documentation appeared to be out of sync with the current workflow code.
I reported the issues to both repositories. After the response from Auto-Monarch by @gmontane (thanks again!), I shared with him an idea to auto-generate the documentation in certain formats, like Markdown, directly from Autosubmit experiment configuration.
Other workflow managers support this already, like Cylc, ecFlow, and cwltool. The idea is to have new attributes in our YAML configuration for documentation. For example:
jobs.yaml
JOBS:
LOCAL_SETUP:
FILE: template/local_setup.sh
RUNNING: once
TITLE: A task to prepare the local archives used by the model
DESCRIPTION: |
This verifies if the expected files exist in the local directory,
`tar`'s these files, and creates a log line about its execution.
Executed once, this step is optional if you have a pre-existing
model binary, and can be disabled with the environment variable
`ABCDE`...
SIM:
FILE: template/ini.sh
TITLE: The model simulation task
DESCRIPTION: |
This is the most important task of this workflow. It will ...
expdef.yaml
DEFAULT:
EXPID: a000
HPCARCH: local
TITLE: Auto-Application
DESCRIPTION: |
Auto-Application is the Autosubmit workflow for the Application model.
It is developed and maintained by the team ABC.
It is a model of this type, that does this, and that...
documentation: <https://intranet...>
issue tracking: <https://gitlab...>
source code: <https://gitlab...>
contact point: [ABC DEF](mailto:....)
In this example, the TITLE
attribute would be a short title for the workflow node, by default the experiment ID or the task ID. The DESCRIPTION
attribute would then be a longer text with links, documentation about the task or workflow, who to contact, what's the expected input, etc.
These values would then be used by, maybe, a new subcommand like autosubmit autodoc
, to produce text, HTML, or Markdown output.
@gmontane also suggested to use the GitLab API to make it easier to update the Wiki when there's a change. I think that would definitely keep the sources and docs in sync!
Another suggestion from @gmontane: “and we should define a common format between projects.”
By getting some feedback from users for this common format, I think we could start simple with a Markdown template using Python stdlib's string module to render it. This way we wouldn't have to add a new dependency to Autosubmit, and we could improve the solution later if needed (but that's more about the implementation, that could be discussed later).