diff --git a/docs/source/agui/._fig1_gui.png b/docs/source/agui/._fig1_gui.png deleted file mode 100644 index b404ad510c2bde06a3e89557449d7e0353ee6ddb..0000000000000000000000000000000000000000 Binary files a/docs/source/agui/._fig1_gui.png and /dev/null differ diff --git a/docs/source/agui/._fig4_gui.jpg b/docs/source/agui/._fig4_gui.jpg deleted file mode 100644 index dfd9b763af80c60532104b529c9d978781d2998e..0000000000000000000000000000000000000000 Binary files a/docs/source/agui/._fig4_gui.jpg and /dev/null differ diff --git a/docs/source/agui/._fig5_gui.jpg b/docs/source/agui/._fig5_gui.jpg deleted file mode 100644 index 6475c7f603376c991a7a897b95f7d01659beb98a..0000000000000000000000000000000000000000 Binary files a/docs/source/agui/._fig5_gui.jpg and /dev/null differ diff --git a/docs/source/agui/._fig_experiment.jpg b/docs/source/agui/._fig_experiment.jpg deleted file mode 100644 index 08c2a053def811e48a77820d0ffe6bd1cbeb2bc7..0000000000000000000000000000000000000000 Binary files a/docs/source/agui/._fig_experiment.jpg and /dev/null differ diff --git a/docs/source/agui/._fig_graph_2.jpg b/docs/source/agui/._fig_graph_2.jpg deleted file mode 100644 index 992c35e9bf51f4eb369a42cd1856fe93be9ebc11..0000000000000000000000000000000000000000 Binary files a/docs/source/agui/._fig_graph_2.jpg and /dev/null differ diff --git a/docs/source/agui/._fig_performance_1.jpg b/docs/source/agui/._fig_performance_1.jpg deleted file mode 100644 index 5d892c20dd15243ad776079c7e230c9e89ae7c23..0000000000000000000000000000000000000000 Binary files a/docs/source/agui/._fig_performance_1.jpg and /dev/null differ diff --git a/docs/source/agui/._fig_stat_1.jpg b/docs/source/agui/._fig_stat_1.jpg deleted file mode 100644 index af51e636ad6af760dcc139c3ac5130562b4bf7e5..0000000000000000000000000000000000000000 Binary files a/docs/source/agui/._fig_stat_1.jpg and /dev/null differ diff --git a/docs/source/agui/._fig_stat_2.jpg b/docs/source/agui/._fig_stat_2.jpg deleted file mode 100644 index 89d78d80f3e1b52648ed05753ad44ef4c4c297af..0000000000000000000000000000000000000000 Binary files a/docs/source/agui/._fig_stat_2.jpg and /dev/null differ diff --git a/docs/source/agui/._fig_tree_2.jpg b/docs/source/agui/._fig_tree_2.jpg deleted file mode 100644 index 3d00ac44d69875e3f16d2410e866e43da886ca9d..0000000000000000000000000000000000000000 Binary files a/docs/source/agui/._fig_tree_2.jpg and /dev/null differ diff --git a/docs/source/agui/fig_experiment.jpg b/docs/source/agui/experiment/fig/fig_experiment.jpg similarity index 100% rename from docs/source/agui/fig_experiment.jpg rename to docs/source/agui/experiment/fig/fig_experiment.jpg diff --git a/docs/source/agui/experiment.rst b/docs/source/agui/experiment/index.rst similarity index 94% rename from docs/source/agui/experiment.rst rename to docs/source/agui/experiment/index.rst index 9881d923ccd79aca78e9b7842c6be8e39c155fcb..55cd49b83aacdcab31fb6cffc8542811445cb875 100644 --- a/docs/source/agui/experiment.rst +++ b/docs/source/agui/experiment/index.rst @@ -5,7 +5,7 @@ Experiment Information This component offers the main information about your experiment. -.. figure:: fig_experiment.jpg +.. figure:: fig/fig_experiment.jpg :name: experiment_view :width: 100% :align: center diff --git a/docs/source/agui/fig1_gui.png b/docs/source/agui/fig/fig1_gui.png similarity index 100% rename from docs/source/agui/fig1_gui.png rename to docs/source/agui/fig/fig1_gui.png diff --git a/docs/source/agui/fig2_gui.png b/docs/source/agui/fig/fig2_gui.png similarity index 100% rename from docs/source/agui/fig2_gui.png rename to docs/source/agui/fig/fig2_gui.png diff --git a/docs/source/agui/fig3_gui.png b/docs/source/agui/fig/fig3_gui.png similarity index 100% rename from docs/source/agui/fig3_gui.png rename to docs/source/agui/fig/fig3_gui.png diff --git a/docs/source/agui/fig4_gui.jpg b/docs/source/agui/fig/fig4_gui.jpg similarity index 100% rename from docs/source/agui/fig4_gui.jpg rename to docs/source/agui/fig/fig4_gui.jpg diff --git a/docs/source/agui/fig5_gui.jpg b/docs/source/agui/fig/fig5_gui.jpg similarity index 100% rename from docs/source/agui/fig5_gui.jpg rename to docs/source/agui/fig/fig5_gui.jpg diff --git a/docs/source/agui/fig_ev_1.png b/docs/source/agui/fig/fig_ev_1.png similarity index 100% rename from docs/source/agui/fig_ev_1.png rename to docs/source/agui/fig/fig_ev_1.png diff --git a/docs/source/agui/fig_tree_1.png b/docs/source/agui/fig/fig_tree_1.png similarity index 100% rename from docs/source/agui/fig_tree_1.png rename to docs/source/agui/fig/fig_tree_1.png diff --git a/docs/source/agui/fig_tree_2.png b/docs/source/agui/fig/fig_tree_2.png similarity index 100% rename from docs/source/agui/fig_tree_2.png rename to docs/source/agui/fig/fig_tree_2.png diff --git a/docs/source/agui/fig_graph_1.jpg b/docs/source/agui/graph/fig/fig_graph_1.jpg similarity index 100% rename from docs/source/agui/fig_graph_1.jpg rename to docs/source/agui/graph/fig/fig_graph_1.jpg diff --git a/docs/source/agui/fig_graph_2.jpg b/docs/source/agui/graph/fig/fig_graph_2.jpg similarity index 100% rename from docs/source/agui/fig_graph_2.jpg rename to docs/source/agui/graph/fig/fig_graph_2.jpg diff --git a/docs/source/agui/fig_graph_3.jpg b/docs/source/agui/graph/fig/fig_graph_3.jpg similarity index 100% rename from docs/source/agui/fig_graph_3.jpg rename to docs/source/agui/graph/fig/fig_graph_3.jpg diff --git a/docs/source/agui/fig_graph_4.jpg b/docs/source/agui/graph/fig/fig_graph_4.jpg similarity index 100% rename from docs/source/agui/fig_graph_4.jpg rename to docs/source/agui/graph/fig/fig_graph_4.jpg diff --git a/docs/source/agui/graph.rst b/docs/source/agui/graph/index.rst similarity index 98% rename from docs/source/agui/graph.rst rename to docs/source/agui/graph/index.rst index 285c60ed54d7833f9270d24439718f170726909d..96e95d5ae9f6e2ba8e5af7922d809566c873e642 100644 --- a/docs/source/agui/graph.rst +++ b/docs/source/agui/graph/index.rst @@ -5,7 +5,7 @@ Graph Representation The Graph Representation of the experiment offers a dependency oriented view. -.. figure:: fig_graph_2.jpg +.. figure:: fig/fig_graph_2.jpg :name: experiment_graph :width: 100% :align: center @@ -55,7 +55,7 @@ Wrappers Representation Wrappers are an important feature of Autosubmit, and as such, it should be possible to visualize them in the graph representation. -.. figure:: fig_graph_3.jpg +.. figure:: fig/fig_graph_3.jpg :name: experiment_graph_wr :width: 100% :align: center @@ -76,7 +76,7 @@ If the experiment is ``RUNNING`` you will see at the top right corner the button Job Search ---------- -.. figure:: fig_graph_4.jpg +.. figure:: fig/fig_graph_4.jpg :name: experiment_graph_search :width: 100% :align: center diff --git a/docs/source/autosubmit-gui.rst b/docs/source/agui/index.rst similarity index 93% rename from docs/source/autosubmit-gui.rst rename to docs/source/agui/index.rst index 67b0fa94aac8f772fcf7d919d9fe72fa3a14fe05..e319633825f5d61542f9a808a5b9907f30c71aab 100644 --- a/docs/source/autosubmit-gui.rst +++ b/docs/source/agui/index.rst @@ -1,6 +1,15 @@ Autosubmit GUI ############## +.. toctree:: + :hidden: + experiment/index + tree/index + graph/index + log/index + performance/index + statistics/index + .. _autosubmitGUIMP: Autosubmit GUI Main Page @@ -12,7 +21,7 @@ Inside the Barcelona Supercomputing Internal Network you can find the latest ver When you enter the site, you will be presented with the following page: -.. figure:: /agui/fig1_gui.png +.. figure:: fig/fig1_gui.png :name: first_page :width: 100% :align: center @@ -22,7 +31,7 @@ When you enter the site, you will be presented with the following page: Here you can search for any ongoing or past experiment by typing some text in the Search input box and pressing **Search**: the search engine will look for coincidences between your input string and any of the description, owner or name of the experiment fields. The results will be shown below ordered by status, experiments ``RUNNING`` will be shown in the first rows. You can also click on the ``Running`` button, and all the experiments that are currently running will be listed. The results will look like: -.. figure:: /agui/fig2_gui.png +.. figure:: fig/fig2_gui.png :name: first_page_search :width: 100% :align: center @@ -32,7 +41,7 @@ Here you can search for any ongoing or past experiment by typing some text in th If you click on ``Show Detailed Data``, summary data for each experiment (result) will be loaded. These are data details from the experiment run, useful to see its status at a glance. Progress bars and status will use different colors to highlight the important information. -.. figure:: /agui/fig3_gui.png +.. figure:: fig/fig3_gui.png :name: first_page_search_plus :width: 100% :align: center @@ -42,7 +51,7 @@ If you click on ``Show Detailed Data``, summary data for each experiment (result For each experiment, you see the following data: -.. figure:: /agui/fig4_gui.jpg +.. figure:: fig/fig4_gui.jpg :name: first_page_search_plus_description :width: 100% :align: center @@ -65,7 +74,7 @@ For each experiment, you see the following data: 13. *Number of Suspended Jobs* 14. *Number of Failed Jobs*: If there are Failed jobs, a list of the names of those jobs will be displayed. -.. figure:: /agui/fig5_gui.jpg +.. figure:: fig/fig5_gui.jpg :name: first_page_search_plus_description_sim :width: 100% :align: center @@ -77,12 +86,5 @@ In experiments that include ``SIM`` jobs, you will also see the average queuing After clicking on the **MORE** button, you will be presented with the **Experiment Page**, which is the main view that Autosubmit provides. These are its main components: -.. toctree:: - agui/experiment - agui/tree - agui/graph - agui/log - agui/performance - agui/statistics .. important:: To improve response times, Autosubmit GUI will try to store the dependency structure of your experiment in a database filed called ``structure_expid.db`` where ``expid`` is the name of your experiment. This file will be located in ``/esarchive/autosubmit/expid/pkl/``. \ No newline at end of file diff --git a/docs/source/agui/fig_log_1.jpg b/docs/source/agui/log/fig/fig_log_1.jpg similarity index 100% rename from docs/source/agui/fig_log_1.jpg rename to docs/source/agui/log/fig/fig_log_1.jpg diff --git a/docs/source/agui/fig_log_2.jpg b/docs/source/agui/log/fig/fig_log_2.jpg similarity index 100% rename from docs/source/agui/fig_log_2.jpg rename to docs/source/agui/log/fig/fig_log_2.jpg diff --git a/docs/source/agui/log.rst b/docs/source/agui/log/index.rst similarity index 94% rename from docs/source/agui/log.rst rename to docs/source/agui/log/index.rst index 1cd40275bb41472fa88e38dea21f571256654488..9b3cc9a886afff145741a6a432a923d0f332ed4e 100644 --- a/docs/source/agui/log.rst +++ b/docs/source/agui/log/index.rst @@ -5,7 +5,7 @@ Autosubmit Log When you click on the ``Log`` tab, you will see the button ``Show Log``: -.. figure:: fig_log_1.jpg +.. figure:: fig/fig_log_1.jpg :name: experiment_log :width: 100% :align: center @@ -17,7 +17,7 @@ When you click on the ``Log`` tab, you will see the button ``Show Log``: When you click on the ``Show Log`` button, the last 150 lines of the log will be displayed: -.. figure:: fig_log_2.jpg +.. figure:: fig/fig_log_2.jpg :name: experiment_log_open :width: 100% :align: center diff --git a/docs/source/agui/fig_performance_1.jpg b/docs/source/agui/performance/fig/fig_performance_1.jpg similarity index 100% rename from docs/source/agui/fig_performance_1.jpg rename to docs/source/agui/performance/fig/fig_performance_1.jpg diff --git a/docs/source/agui/performance.rst b/docs/source/agui/performance/index.rst similarity index 89% rename from docs/source/agui/performance.rst rename to docs/source/agui/performance/index.rst index ab34739ad4f197165c091dc1d15712fd9448dc2e..906e40733a33e43207edb907f4fb701ad64d534a 100644 --- a/docs/source/agui/performance.rst +++ b/docs/source/agui/performance/index.rst @@ -5,7 +5,7 @@ Performance Metrics The Performance Metrics tabs offers a set of metrics that try to measure the efficiency of the simulation performed, and other aspects of it. -.. figure:: fig_performance_1.jpg +.. figure:: fig/fig_performance_1.jpg :name: experiment_performance :width: 100% :align: center diff --git a/docs/source/agui/fig_stat_1.jpg b/docs/source/agui/statistics/fig/fig_stat_1.jpg similarity index 100% rename from docs/source/agui/fig_stat_1.jpg rename to docs/source/agui/statistics/fig/fig_stat_1.jpg diff --git a/docs/source/agui/fig_stat_2.jpg b/docs/source/agui/statistics/fig/fig_stat_2.jpg similarity index 100% rename from docs/source/agui/fig_stat_2.jpg rename to docs/source/agui/statistics/fig/fig_stat_2.jpg diff --git a/docs/source/agui/statistics.rst b/docs/source/agui/statistics/index.rst similarity index 90% rename from docs/source/agui/statistics.rst rename to docs/source/agui/statistics/index.rst index 1756dabbb68cac8a36a889149b472e7ca1fb7447..7321f125d4838a21a4172e227c3d932fa3a269ad 100644 --- a/docs/source/agui/statistics.rst +++ b/docs/source/agui/statistics/index.rst @@ -6,7 +6,7 @@ Autosubmit Statistics When you click on the ``Statistics`` tab, you will see two input boxes: ``Section`` and ``Hours``, followed by the button ``Get Statistics``: -.. figure:: fig_stat_1.jpg +.. figure:: fig/fig_stat_1.jpg :name: experiment_stats :width: 100% :align: center @@ -18,7 +18,7 @@ There is also a brief explanation of the input fields and expected result. Basic In this example we have queried 3 hours into the past: -.. figure:: fig_stat_2.jpg +.. figure:: fig/fig_stat_2.jpg :name: experiment_stats_open :width: 100% :align: center diff --git a/docs/source/agui/fig_tree_2.jpg b/docs/source/agui/tree/fig/fig_tree_2.jpg similarity index 100% rename from docs/source/agui/fig_tree_2.jpg rename to docs/source/agui/tree/fig/fig_tree_2.jpg diff --git a/docs/source/agui/tree.rst b/docs/source/agui/tree/index.rst similarity index 99% rename from docs/source/agui/tree.rst rename to docs/source/agui/tree/index.rst index 1ffaba0955d9ca5e67ea8955d0ac0c23cc897275..ff9e604acf5882e45b3c655e83239eb3134d8695 100644 --- a/docs/source/agui/tree.rst +++ b/docs/source/agui/tree/index.rst @@ -5,7 +5,7 @@ Tree Representation The Tree Representation offers a structured view of the experiment. -.. figure:: fig_tree_2.jpg +.. figure:: fig/fig_tree_2.jpg :name: experiment_tree :width: 100% :align: center diff --git a/docs/source/project.rst b/docs/source/devguide/index.rst similarity index 92% rename from docs/source/project.rst rename to docs/source/devguide/index.rst index 6c91991c14cbe5dea27b65c4db4f0d5e096aa437..eb281906ef2e05513b7df70004a9f1d44c0e1af3 100644 --- a/docs/source/project.rst +++ b/docs/source/devguide/index.rst @@ -1,6 +1,6 @@ -#################### -Developing a project -#################### +############################## +Developing an EC-Earth Project +############################## Autosubmit is used at BSC to run EC-Earth. To do that, a git repository has been created that contains the model source code and the scripts used to run the tasks. diff --git a/docs/source/variables.rst b/docs/source/devguide/variables.rst similarity index 100% rename from docs/source/variables.rst rename to docs/source/devguide/variables.rst diff --git a/docs/source/faq-original.rst b/docs/source/faq-original.rst deleted file mode 100644 index 4db1c2b004f6e8ad00f0e64e61664c8a63305243..0000000000000000000000000000000000000000 --- a/docs/source/faq-original.rst +++ /dev/null @@ -1,6 +0,0 @@ -############################## -Frequent Questions and Answers -############################## - -The latest version of **Autosubmit** implements a code system that guides you through the process of fixing some of the common problems you might find. Consequently, the **FAQ** section has been replaced by :ref:`faqnew`, where you will find the list of error codes, their descriptions, and solutions. - diff --git a/docs/source/index.rst b/docs/source/index.rst index 35d31fb76a43148168e50d347efcdcd34e2abb90..9ba748e0dc0827c9e3adde8d280b3db157cce256 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -8,19 +8,74 @@ Welcome to autosubmit's documentation! ###################################### .. toctree:: + :maxdepth: 1 + :hidden: + + /introduction/index + +.. toctree:: + :caption: Quick Start Guide + :maxdepth: 1 + :hidden: + + /qstartguide/index + +.. toctree:: + :caption: Installation + :maxdepth: 1 + :hidden: + + /installation/index + +.. toctree:: + :caption: User Guide :maxdepth: 2 + :hidden: + + /userguide/create/index + /userguide/configure/index + /userguide/defining workflows/index + /userguide/wrappers/index.rst + /userguide/run/index + /userguide/modifying workflows/index.rst + /userguide/manage/index + /userguide/monitor and check/index + +.. toctree:: + :caption: Developer Guide + :maxdepth: 1 + :hidden: + + /devguide/index + +.. toctree:: + :caption: Troubleshooting + :maxdepth: 1 + :hidden: + + /troubleshooting/index + /troubleshooting/error-codes + +.. toctree:: + :caption: Module Documentation + :maxdepth: 1 + :hidden: + + /moduledoc/index + - introduction - tutorial - devel_proj - installation - usage - workflows - faq-original - troubleshoot - faq - project - variables - codedoc/main - autosubmit-gui +Autosubmit is a Python software to manage complicated workflows on HPC platforms. +Automatization + Autosubmit manages job submission and dependencies without user intervention +Data Provenance. + Autosubmit keeps tracks of data generated by each experiment by assigning to them + unique ids. +Failure Tolerance + Autosubmit manages automatic retrials and has the ability to rerun specific parts of + the experiment in case of failure +Resource Management + Autosubmit supports a per-platform configuration, allowing users to run their experiments + without adapting job scripts. +Multiple Platform + Autosubmit can run jobs of an experiment in different platforms \ No newline at end of file diff --git a/docs/source/installation.rst b/docs/source/installation/index.rst similarity index 94% rename from docs/source/installation.rst rename to docs/source/installation/index.rst index 2b4a654971c9a2c6228f3563062d81add6b3c0ee..6573f8723cd789594842c6bfebfcd71d76c50e35 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation/index.rst @@ -1,9 +1,6 @@ -############ -Installation -############ - -How to install -============== +######################### +How to Install Autosubmit +######################### The Autosubmit code is maintained in *PyPi*, the main source for python packages. @@ -38,8 +35,76 @@ or download, unpack and: .. hint:: To see the changelog, use ``autosubmit changelog`` +Examples +======== + +Sequence of instructions to install Autosubmit and its dependencies in Ubuntu. +------------------------------------------------------------------------------ + +.. code-block:: bash + + + # Update repositories + apt update + + # Avoid interactive stuff + export DEBIAN_FRONTEND=noninteractive + + # Dependencies + apt install wget curl python2 python-tk python2-dev graphviz -y -q + + # Additional dependencies related with pycrypto + apt install build-essential libssl-dev libffi-dev -y -q + + # Download get pip script and launch it + wget https://bootstrap.pypa.io/pip/2.7/get-pip.py + python2 get-pip.py + + # Install autosubmit using pip + pip2 install autosubmit + + # Check that we can execute autosubmit commands + autosubmit -h + + # Configure + autosubmit configure + + # Install + autosubmit install + + # Get expid + autosubmit expid -H TEST -d "Test exp." + + # Create with -np + # Since it was a new install the expid will be a000 + autosubmit create a000 -np + +Sequence of instructions to install Autosubmit and its dependencies with conda. +------------------------------------------------------------------------------- + +.. code-block:: bash + + # Download conda + wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh./Miniconda3-py39_4.12.0-Linux-x86_64.sh + # Launch it + ./Miniconda3-py39_4.12.0-Linux-x86_64.sh + # Download git + apt install git -y -q + # Download autosubmit + git clone https://earth.bsc.es/gitlab/es/autosubmit.git -b v3.14.0 + cd autosubmit + # Create conda environment + conda env update -f environment.yml -n autosubmit python=2 + # Activate env + source activate autosubmit + # Test autosubmit + autosubmit -v + # Configure autosubmitrc and install database as indicated in this doc + + +################ How to configure -================ +################ After installation, you have to configure database and path for Autosubmit. In order to use the default settings, just create a directory called `autosubmit` in your home directory before running the configure command. @@ -71,11 +136,9 @@ For installing the database for Autosubmit on the configured folder, when no dat autosubmit install -.. danger:: Be careful ! autosubmit install will create a blank database. +.. important:: Be careful ! autosubmit install will create a blank database. -Lastly, if autosubmit configure doesn't work for you or you need to configure additional info create: - -Create or modify /etc/autosubmitrc file or ~/.autosubmitrc with the information as follows: +Lastly, if autosubmit configure doesn't work for you or you need to configure additional info create or modify /etc/autosubmitrc file or ~/.autosubmitrc with the information as follows: .. code-block:: ini @@ -119,71 +182,4 @@ From 3.14+ onwards, autosubmit commands can be tailored to run on specific machi * If no commands are defined, all commands are authorized. * If no machines are defined, all machines are authorized. -Now you are ready to use Autosubmit ! - - -Examples -======== - -Sequence of instructions to install Autosubmit and its dependencies in Ubuntu. ------------------------------------------------------------------------------- - -.. code-block:: bash - - - # Update repositories - apt update - - # Avoid interactive stuff - export DEBIAN_FRONTEND=noninteractive - - # Dependencies - apt install wget curl python2 python-tk python2-dev graphviz -y -q - - # Additional dependencies related with pycrypto - apt install build-essential libssl-dev libffi-dev -y -q - - # Download get pip script and launch it - wget https://bootstrap.pypa.io/pip/2.7/get-pip.py - python2 get-pip.py - - # Install autosubmit using pip - pip2 install autosubmit - - # Check that we can execute autosubmit commands - autosubmit -h - - # Configure - autosubmit configure - - # Install - autosubmit install - - # Get expid - autosubmit expid -H TEST -d "Test exp." - - # Create with -np - # Since it was a new install the expid will be a000 - autosubmit create a000 -np - -Sequence of instructions to install Autosubmit and its dependencies with conda. -------------------------------------------------------------------------------- - -.. code-block:: bash - - # Download conda - wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh./Miniconda3-py39_4.12.0-Linux-x86_64.sh - # Launch it - ./Miniconda3-py39_4.12.0-Linux-x86_64.sh - # Download git - apt install git -y -q - # Download autosubmit - git clone https://earth.bsc.es/gitlab/es/autosubmit.git -b v3.14.0 - cd autosubmit - # Create conda environment - conda env update -f environment.yml -n autosubmit python=2 - # Activate env - source activate autosubmit - # Test autosubmit - autosubmit -v - # Configure autosubmitrc and install database as indicated in this doc +Now you are ready to use Autosubmit ! \ No newline at end of file diff --git a/docs/source/fig1.png b/docs/source/introduction/fig1.png similarity index 100% rename from docs/source/fig1.png rename to docs/source/introduction/fig1.png diff --git a/docs/source/fig2.png b/docs/source/introduction/fig2.png similarity index 100% rename from docs/source/fig2.png rename to docs/source/introduction/fig2.png diff --git a/docs/source/fig3.png b/docs/source/introduction/fig3.png similarity index 100% rename from docs/source/fig3.png rename to docs/source/introduction/fig3.png diff --git a/docs/source/introduction.rst b/docs/source/introduction/index.rst similarity index 74% rename from docs/source/introduction.rst rename to docs/source/introduction/index.rst index 5fbe85179ef85949ebcee7b31ff6a4c12fc27a26..b39eee54145491a64a16c606c0578de9d49663d9 100644 --- a/docs/source/introduction.rst +++ b/docs/source/introduction/index.rst @@ -5,14 +5,15 @@ Introduction What is Autosubmit ? ==================== -:: +Autosubmit is a python-based workflow manager to create, manage and monitor experiments by using Computing Clusters, HPC’s and Supercomputers remotely via ssh. It has support for experiments running in more than one HPC and for different workflow configurations. + +It is currently used at Barcelona Supercomputing Centre (BSC) to run models (EC-Earth, MONARCH, NEMO, CALIOPE, HERMES...), operational toolchains (S2S4E), data-download workflows (ECMWF MARS), and many other. - Autosubmit is a python-based workflow manager to create, manage and monitor experiments by using Computing Clusters, HPC’s and Supercomputers remotely via ssh. It has support for experiments running in more than one HPC and for different workflow configurations. - Autosubmit is currently used at Barcelona Supercomputing Centre (BSC) to run models (EC-Earth, MONARCH, NEMO, CALIOPE, HERMES...), operational toolchains (S2S4E), data-download workflows (ECMWF MARS), and many other. - Autosubmit has been used to manage models running at supercomputers in BSC, ECMWF, IC3, CESGA, EPCC, PDC and OLCF. - Autosubmit is now available via PyPi package under the terms of GNU General Public License. +Autosubmit has been used to manage models running at supercomputers in BSC, ECMWF, IC3, CESGA, EPCC, PDC and OLCF. +Autosubmit is now available via PyPi package under the terms of GNU General Public License. +Moreover, Autosubmit has a user-friendly, web based, and fresh GUI Get involved or contact us: @@ -27,7 +28,7 @@ Why is Autosubmit needed ? Autosubmit is the only existing tool that satisfies the following requirements from the weather and climate community: -- **Automatisation** Job submission to machines and dependencies between jobs are managed by Autosubmit. No user intervention is needed. +- **Automatization** Job submission to machines and dependencies between jobs are managed by Autosubmit. No user intervention is needed. - **Data provenance** Assigns unique identifiers for each experiment and stores information about model version, experiment configuration and computing facilities used in the whole process. - **Failure tolerance** Automatic retrials and ability to rerun chunks in case of corrupted or missing data. - **Resource management** Autosubmit manages supercomputer particularities, allowing users to run their experiments in the available machine without having to adapt the code. Autosubmit also allows to submit tasks from the same experiment to different platforms. @@ -59,7 +60,9 @@ To create a new experiment, run the command: *HPCname* is the name of the main HPC platform for the experiment: it will be the default platform for the tasks. *Description* is a brief experiment description. -This command assigns a unique four character identifier (``xxxx``, names starting from a letter, the other three characters) to the experiment and creates a new folder in experiments repository with structure shown in Figure :numref:`exp_folder`. +This command assigns to the experiment a unique four alphanumerical characters identifier, where the first has reserved letters *o* and +*t*. It then creates a new folder in experiments repository with structure shown in Figure :numref:`exp_folder`. + .. figure:: fig1.png :name: exp_folder @@ -100,13 +103,13 @@ To run the experiment, just execute the command: Autosubmit will start submitting jobs to the relevant platforms (both HPC and supporting computers) by using the scripts specified in ``jobs_xxxx.conf``. Autosubmit will substitute variables present on scripts where handlers appear in *%variable_name%* format. Autosubmit provides variables for *current chunk*, *start date*, *member*, *computer configuration* and more, and also will replace variables form ``proj_xxxx.conf``. -To monitor the status of the experiment, the command: +To monitor the status of the experiment, issue the command: :: autosubmit monitor xxxx -is available. This will plot the workflow of the experiment and the current status. +This will plot the workflow of the experiment and the current status. .. figure:: fig3.png :width: 70% diff --git a/docs/source/codedoc/autosubmit.rst b/docs/source/moduledoc/autosubmit.rst similarity index 100% rename from docs/source/codedoc/autosubmit.rst rename to docs/source/moduledoc/autosubmit.rst diff --git a/docs/source/codedoc/config.rst b/docs/source/moduledoc/config.rst similarity index 100% rename from docs/source/codedoc/config.rst rename to docs/source/moduledoc/config.rst diff --git a/docs/source/codedoc/database.rst b/docs/source/moduledoc/database.rst similarity index 100% rename from docs/source/codedoc/database.rst rename to docs/source/moduledoc/database.rst diff --git a/docs/source/codedoc/git.rst b/docs/source/moduledoc/git.rst similarity index 100% rename from docs/source/codedoc/git.rst rename to docs/source/moduledoc/git.rst diff --git a/docs/source/codedoc/main.rst b/docs/source/moduledoc/index.rst similarity index 55% rename from docs/source/codedoc/main.rst rename to docs/source/moduledoc/index.rst index 737c0d767bc130ca400a67edf718f541987abb4f..024ec6774186c619435423f3d54b41341fe04a5f 100644 --- a/docs/source/codedoc/main.rst +++ b/docs/source/moduledoc/index.rst @@ -1,6 +1,6 @@ -******************** -Module documentation -******************** +******* +Modules +******* .. toctree:: :titlesonly: diff --git a/docs/source/codedoc/job.rst b/docs/source/moduledoc/job.rst similarity index 100% rename from docs/source/codedoc/job.rst rename to docs/source/moduledoc/job.rst diff --git a/docs/source/codedoc/monitor.rst b/docs/source/moduledoc/monitor.rst similarity index 100% rename from docs/source/codedoc/monitor.rst rename to docs/source/moduledoc/monitor.rst diff --git a/docs/source/codedoc/platforms.rst b/docs/source/moduledoc/platforms.rst similarity index 100% rename from docs/source/codedoc/platforms.rst rename to docs/source/moduledoc/platforms.rst diff --git a/docs/source/workflows/dummy.png b/docs/source/qstartguide/dummy.png similarity index 100% rename from docs/source/workflows/dummy.png rename to docs/source/qstartguide/dummy.png diff --git a/docs/source/tutorial.rst b/docs/source/qstartguide/index.rst similarity index 88% rename from docs/source/tutorial.rst rename to docs/source/qstartguide/index.rst index ef8691cbacd085885bce4f5bb53d3ae64e5a0d0a..6ec8898bdf0ed311e223be49a2cc59bdb85d9f86 100644 --- a/docs/source/tutorial.rst +++ b/docs/source/qstartguide/index.rst @@ -3,9 +3,9 @@ Tutorial start guide ==================== -This tutorial is a starter’s guide to running a dummy experiment with Autosubmit. +This tutorial is a starter’s guide to run a dummy experiment with Autosubmit. -Dummy experiments run workflows with non-expensive empty tasks and therefore are ideal for teaching and testing purposes. +Dummy experiments run workflows with inexpensive empty tasks and therefore are ideal for teaching and testing purposes. Real experiments instead run workflows with complex tasks. To read information about how to develop parameterizable tasks for Autosubmit workflows, refer to :ref:`develproject`. @@ -16,7 +16,7 @@ Autosubmit needs to establish **password-less SSH connections** in order to run Ensure that you have a **password-less** connection to all platforms you want to use in your experiment. If you are unsure how to do this, please follow these instructions: -- Open a terminal and prompt ```ssh-keygen -t rsa -b 4096 -C "email@email.com" -m PEM``` +- Open a terminal and prompt ``ssh-keygen -t rsa -b 4096 -C "email@email.com" -m PEM`` - Copy the resulting key to your platform of choice. Via SCP or ssh-copy-key. Description of most used commands @@ -75,11 +75,11 @@ The output of the command will show the expid of the experiment and generate the - User scripts and project code. (empty) -Then, prompt ``autosubmit create -np`` and Autosubmit will generate the workflow graph. +Then, execute ``autosubmit create -np`` and Autosubmit will generate the workflow graph. Run and monitoring: =============== - To run an experiment, use ```autosubmit run ```. Autosubmit run experiments performing the following operations: + To run an experiment, use ```autosubmit run ```. Autosubmit runs experiments performing the following operations: - First, it **checks the experiment configuration**. If it is wrong, it won't proceed further. - Second, it **runs the experiment while retrieving all logs** from completed or failed tasks as they run. @@ -87,7 +87,7 @@ Run and monitoring: While the experiment is running, it can be visualized via ``autosubmit monitor ``. -.. figure:: workflows/dummy.png +.. figure:: dummy.png :name: dummy_workflow :width: 100% :align: center @@ -96,9 +96,9 @@ While the experiment is running, it can be visualized via ``autosubmit monitor < illustrates the output of the autosubmit monitor. It describes all workflow jobs' possible status and actual status. -At the same time, the ``/tmp`` gets filled with the cmd scripts generated by Autosubmit to run the local and remote tasks (in this case, they are sent and submitted to the remote platform(s)). +Concurrently, the ``/tmp`` gets filled with the cmd scripts generated by Autosubmit to run the local and remote tasks (in this case, they are sent and submitted to the remote platform(s)). -On the other hand, the ``ASLOGS`` and ``LOG_a000`` folders are filling up with AS command logs and jobs logs. +Autosubmit keeps logs at ``ASLOGS`` and ``LOG_a000`` folders, which are filled up with Autosubmit's command logs and job logs. Configuration summary: ================== @@ -142,7 +142,7 @@ Configuration summary: Final step: Modify and run ========================== - It is time to look into the configuration files of the dummy experiment and modify them with a remote platform to run a workflow with a few more chunks. + It is time to look into the configuration files of the dummy experiment and modify them with a remote platform to run a workflow with a few more chunks. Open expdef.conf @@ -179,4 +179,4 @@ Now open platforms.conf. Note: This will be an example for marenostrum4 SERIAL_QUEUE = debug QUEUE = debug -``autosubmit create ** (without -np)`` will generate the new workflow and ``autosubmit run `` will run the experiment with the latest changes. \ No newline at end of file +``autosubmit create ** (without -np)`` will generate the new workflow and ``autosubmit run `` will run the experiment with the latest changes. diff --git a/docs/source/faq.rst b/docs/source/troubleshooting/error-codes.rst similarity index 99% rename from docs/source/faq.rst rename to docs/source/troubleshooting/error-codes.rst index 7d1e31b3403dd94f88cfe69c1e21911f01f1759c..c92ba38ada9432a57ca1e66e867831decfcba0bf 100644 --- a/docs/source/faq.rst +++ b/docs/source/troubleshooting/error-codes.rst @@ -1,8 +1,8 @@ -.. _faqnew: +.. _errorcodes: -################################## +######################### Error codes and solutions -################################## +######################### Experiment Locked - Critical Error 7000 =================================================== diff --git a/docs/source/troubleshoot.rst b/docs/source/troubleshooting/index.rst similarity index 87% rename from docs/source/troubleshoot.rst rename to docs/source/troubleshooting/index.rst index a36f1f36d1c6a7572286bb2aa88a697f8b35774e..7416539975b04ce76145d074db2a25be80298a4e 100644 --- a/docs/source/troubleshoot.rst +++ b/docs/source/troubleshooting/index.rst @@ -1,6 +1,6 @@ -############### -Troubleshooting -############### +### +FAQ +### How to change the job status stopping autosubmit ================================================ @@ -42,7 +42,7 @@ Other possible errors **I see the `database malformed` error on my experiment log.** -*Explanation*: The latest version of autosubmit uses a database to efficiently track changes in the jobs of your experiment. It might happen that this small database gets corrupted. +*Explanation*: The latest version of autosubmit uses a database to efficiently track changes in the jobs of your experiment. It could have happened that this small database got corrupted. *Solution*: run `autosubmit dbfix expid` where `expid` is the identifier of your experiment. This function will rebuild the database saving as much information as possible (usually all of it). @@ -50,3 +50,7 @@ Other possible errors *Solution*: run `autosubmit pklfix expid`, it will restore the `backup` file if possible. +Error codes +=========== + +The latest version of **Autosubmit** implements a code system that guides you through the process of fixing some of the common problems you might find. Check :ref:`errorcodes`, where you will find the list of error codes, their descriptions, and solutions. \ No newline at end of file diff --git a/docs/source/workflows/group_chunk.png b/docs/source/unused_figs/group_chunk.png similarity index 100% rename from docs/source/workflows/group_chunk.png rename to docs/source/unused_figs/group_chunk.png diff --git a/docs/source/workflows/horizontal-vertical.png b/docs/source/unused_figs/horizontal-vertical.png similarity index 100% rename from docs/source/workflows/horizontal-vertical.png rename to docs/source/unused_figs/horizontal-vertical.png diff --git a/docs/source/workflows/wrapper.png b/docs/source/unused_figs/wrapper.png similarity index 100% rename from docs/source/workflows/wrapper.png rename to docs/source/unused_figs/wrapper.png diff --git a/docs/source/workflows/wrapper_expression.png b/docs/source/unused_figs/wrapper_expression.png similarity index 100% rename from docs/source/workflows/wrapper_expression.png rename to docs/source/unused_figs/wrapper_expression.png diff --git a/docs/source/workflows/wrapper_hybrid.png b/docs/source/unused_figs/wrapper_hybrid.png similarity index 100% rename from docs/source/workflows/wrapper_hybrid.png rename to docs/source/unused_figs/wrapper_hybrid.png diff --git a/docs/source/usage/advanced_features/custom_header.rst b/docs/source/usage/advanced_features/custom_header.rst deleted file mode 100644 index 463d8f2bf6ec85ab8acb37dc76387077ed5bfea6..0000000000000000000000000000000000000000 --- a/docs/source/usage/advanced_features/custom_header.rst +++ /dev/null @@ -1,103 +0,0 @@ -How to set a custom interpreter for your job -============================================ - -If the remote platform does not implement the interpreter you need, you can customize the ``shebang`` of your job script so it points to the relative path of the interpreter you want. - -In the file: - -:: - - vi /cxxx/conf/jobs_cxxx.conf - -.. code-block:: ini - - # Example job with all options specified - - ## Job name - # [JOBNAME] - ## Script to execute. If not specified, job will be omitted from workflow. - ## Path relative to the project directory - # FILE = - ## Platform to execute the job. If not specified, defaults to HPCARCH in expedf file. - ## LOCAL is always defined and refers to current machine - # PLATFORM = - ## Queue to add the job to. If not specified, uses PLATFORM default. - # QUEUE = - ## Defines dependencies from job as a list of parents jobs separated by spaces. - ## Dependencies to jobs in previous chunk, member o startdate, use -(DISTANCE) - # DEPENDENCIES = INI SIM-1 CLEAN-2 - ## Define if jobs runs once, once per stardate, once per member or once per chunk. Options: once, date, member, chunk. - ## If not specified, defaults to once - # RUNNING = once - ## Specifies that job has only to be run after X dates, members or chunk. A job will always be created for the last - ## If not specified, defaults to 1 - # FREQUENCY = 3 - ## On a job with FREQUENCY > 1, if True, the dependencies are evaluated against all - ## jobs in the frequency interval, otherwise only evaluate dependencies against current - ## iteration. - ## If not specified, defaults to True - # WAIT = False - ## Defines if job is only to be executed in reruns. If not specified, defaults to false. - # RERUN_ONLY = False - ## Wallclock to be submitted to the HPC queue in format HH:MM - # WALLCLOCK = 00:05 - ## Processors number to be submitted to the HPC. If not specified, defaults to 1. - ## Wallclock chunk increase (WALLCLOCK will be increased according to the formula WALLCLOCK + WCHUNKINC * (chunk - 1)). - ## Ideal for sequences of jobs that change their expected running time according to the current chunk. - # WCHUNKINC = 00:01 - # PROCESSORS = 1 - ## Threads number to be submitted to the HPC. If not specified, defaults to 1. - # THREADS = 1 - ## Tasks number to be submitted to the HPC. If not specified, defaults to 1. - # Tasks = 1 - ## Enables hyper-threading. If not specified, defaults to false. - # HYPERTHREADING = false - ## Memory requirements for the job in MB - # MEMORY = 4096 - ## Number of retrials if a job fails. If not specified, defaults to the value given on experiment's autosubmit.conf - # RETRIALS = 4 - ## Allows to put a delay between retries, of retrials if a job fails. If not specified, it will be static - # The ideal is to use the +(number) approach or plain(number) in case that the hpc platform has little issues or the experiment will run for a short period of time - # And *(10) in case that the filesystem is having large delays or the experiment will run for a lot of time. - # DELAY_RETRY_TIME = 11 - # DELAY_RETRY_TIME = +11 # will wait 11 + number specified - # DELAY_RETRY_TIME = *11 # will wait 11,110,1110,11110...* by 10 to prevent a too big number - ## Some jobs can not be checked before running previous jobs. Set this option to false if that is the case - # CHECK = False - ## Select the interpreter that will run the job. Options: bash, python, r Default: bash - # TYPE = bash - ## Specify the path to the interpreter. If empty, use system default based on job type . Default: empty - # EXECUTABLE = /my_python_env/python3 - -You can give a path to the ``EXECUTABLE`` setting of your job. Autosubmit will replace the ``shebang`` with the path you provided. - -Example: - -.. code-block:: ini - - [POST] - FILE = POST.sh - DEPENDENCIES = SIM - RUNNING = chunk - WALLCLOCK = 00:05 - EXECUTABLE = /my_python_env/python3 - -This job will use the python interpreter located in the relative path ``/my_python_env/python3/`` - -It is also possible to use variables in the ``EXECUTABLE`` path. - -Example: - -.. code-block:: ini - - [POST] - FILE = POST.sh - DEPENDENCIES = SIM - RUNNING = chunk - WALLCLOCK = 00:05 - EXECUTABLE = %PROJDIR%/my_python_env/python3 - -The result is a ``shebang`` line ``#!/esarchive/autosubmit/my_python_env/python3``. - - - diff --git a/docs/source/usage/advanced_features/index.rst b/docs/source/usage/advanced_features/index.rst deleted file mode 100644 index 143183a801d367131abf886b8c40fcaf3e4d6629..0000000000000000000000000000000000000000 --- a/docs/source/usage/advanced_features/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -.. _advanced_features: - -How to use advanced features -============================ - -.. toctree:: - - migrate - custom_header diff --git a/docs/source/usage/advanced_features/migrate.rst b/docs/source/usage/advanced_features/migrate.rst deleted file mode 100644 index eb26e1f971dce43ce5da11a36f2335f018f47d3f..0000000000000000000000000000000000000000 --- a/docs/source/usage/advanced_features/migrate.rst +++ /dev/null @@ -1,49 +0,0 @@ -How to migrate an experiment -============================ -To migrate an experiment from one user to another, you need to add two parameters for each platform in the platforms configuration file: - - * USER_TO = # Mandatory - * TEMP_DIR = # Mandatory, can be left empty if there are no files on that platform - * SAME_USER = false|true # Default False - - * PROJECT_TO = # Optional, if not specified project will remain the same - * HOST_TO = # Optional, avoid alias if possible, try use direct ip. - - -.. warning:: The USER_TO must be a different user , in case you want to maintain the same user, put SAME_USER = True. - -.. warning:: The temporary directory must be readable by both users (old owner and new owner) - Example for a RES account to BSC account the tmp folder must have rwx|rwx|--- permissions. - The temporary directory must be in the same filesystem. - -User A, To offer the experiment: -:: - - autosubmit migrate --offer expid - -Local files will be archived and remote files put in the HPC temporary directory. - -User A To only offer the remote files -:: - - autosubmit migrate expid --offer --onlyremote - -Only remote files will be put in the HPC temporary directory. - -.. warning:: Be sure that there is no folder named as the expid before do the pick. - The old owner might need to remove temporal files and archive. - To Run the experiment the queue may need to be change. - -.. warning:: If onlyremote option is selected, the pickup must maintain the flag otherwise the command will fail. - -Now to pick the experiment, the user B, must do -:: - - autosubmit migrate --pickup expid - -Local files will be unarchived and remote files copied from the temporal location. - -To only pick the remote files, the user B, must do -:: - - autosubmit migrate --pickup expid --onlyremote diff --git a/docs/source/usage/archive/archive.rst b/docs/source/usage/archive/archive.rst deleted file mode 100644 index 653aed590e41ce21554722480d98e85ab0b530a3..0000000000000000000000000000000000000000 --- a/docs/source/usage/archive/archive.rst +++ /dev/null @@ -1,34 +0,0 @@ -How to archive an experiment -============================ - -To archive the experiment, use the command: -:: - - autosubmit archive EXPID - -*EXPID* is the experiment identifier. - -.. warning:: this command calls implicitly the clean command. Check clean command documentation. - -.. warning:: experiment will be unusable after archiving. If you want to use it, you will need to call first the - unarchive command - - -Options: -:: - - usage: autosubmit archive [-h] expid - - expid experiment identifier - - -h, --help show this help message and exit - - -Example: -:: - - autosubmit archive cxxx - -.. hint:: Archived experiment will be stored as a tar.gz file on a folder named after the year of the last - COMPLETED file date. If not COMPLETED file is present, it will be stored in the folder matching the - date at the time the archive command was run. \ No newline at end of file diff --git a/docs/source/usage/archive/clean.rst b/docs/source/usage/archive/clean.rst deleted file mode 100644 index 35a1c75727318e88a6ca77103eb7c48462819cb6..0000000000000000000000000000000000000000 --- a/docs/source/usage/archive/clean.rst +++ /dev/null @@ -1,44 +0,0 @@ -How to clean the experiment -=========================== - -This procedure allows you to save space after finalising an experiment. -You must execute: -:: - - autosubmit clean EXPID - - -Options: -:: - - usage: autosubmit clean [-h] [-pr] [-p] [-s] expid - - expid experiment identifier - - -h, --help show this help message and exit - -pr, --project clean project - -p, --plot clean plot, only 2 last will remain - -s, --stats clean stats, only last will remain - -* The -p and -s flag are used to clean our experiment ``plot`` folder to save disk space. Only the two latest plots will be kept. Older plots will be removed. - -Example: -:: - - autosubmit clean cxxx -p - -* The -pr flag is used to clean our experiment ``proj`` locally in order to save space (it could be particularly big). - -.. caution:: Bear in mind that if you have not synchronized your experiment project folder with the information available on the remote repository (i.e.: commit and push any changes we may have), or in case new files are found, the clean procedure will be failing although you specify the -pr option. - -Example: -:: - - autosubmit clean cxxx -pr - -A bare copy (which occupies less space on disk) will be automatically made. - -.. hint:: That bare clone can be always reconverted in a working clone if we want to run again the experiment by using ``git clone bare_clone original_clone``. - -.. note:: In addition, every time you run this command with -pr option, it will check the commit unique identifier for local working tree existing on the ``proj`` directory. - In case that commit identifier exists, clean will register it to the ``expdef_cxxx.conf`` file. diff --git a/docs/source/usage/archive/delete.rst b/docs/source/usage/archive/delete.rst deleted file mode 100644 index 5a5b9fec0d025eb2d61563f387c770cd1a5f9028..0000000000000000000000000000000000000000 --- a/docs/source/usage/archive/delete.rst +++ /dev/null @@ -1,30 +0,0 @@ -How to delete the experiment -============================ - -To delete the experiment, use the command: -:: - - autosubmit delete EXPID - -*EXPID* is the experiment identifier. - -.. warning:: DO NOT USE THIS COMMAND IF YOU ARE NOT SURE ! - It deletes the experiment from database and experiment’s folder. - -Options: -:: - - usage: autosubmit delete [-h] [-f] expid - - expid experiment identifier - - -h, --help show this help message and exit - -f, --force deletes experiment without confirmation - - -Example: -:: - - autosubmit delete cxxx - -.. warning:: Be careful ! force option does not ask for your confirmation. diff --git a/docs/source/usage/archive/index.rst b/docs/source/usage/archive/index.rst deleted file mode 100644 index edf15bb1e34b80c22331fa2a6138876e2601e289..0000000000000000000000000000000000000000 --- a/docs/source/usage/archive/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -.. _archive: - -How to archive and clean an experiment -====================================== - -.. toctree:: - - clean - delete - archive - unarchive diff --git a/docs/source/usage/archive/unarchive.rst b/docs/source/usage/archive/unarchive.rst deleted file mode 100644 index d05204494b87b2635b34ec2e8d13e12050107c7e..0000000000000000000000000000000000000000 --- a/docs/source/usage/archive/unarchive.rst +++ /dev/null @@ -1,25 +0,0 @@ -How to unarchive an experiment -============================== - -To unarchive an experiment, use the command: -:: - - autosubmit unarchive EXPID - -*EXPID* is the experiment identifier. - -Options: -:: - - usage: autosubmit unarchive [-h] expid - - expid experiment identifier - - -h, --help show this help message and exit - - -Example: -:: - - autosubmit unarchive cxxx - diff --git a/docs/source/usage/configuration/check.rst b/docs/source/usage/configuration/check.rst deleted file mode 100644 index a15a648f098067d490fa7ebe5a971e92ade6c30d..0000000000000000000000000000000000000000 --- a/docs/source/usage/configuration/check.rst +++ /dev/null @@ -1,80 +0,0 @@ -How to check the experiment configuration -========================================= -To check the configuration of the experiment, use the command: -:: - - autosubmit check EXPID - -*EXPID* is the experiment identifier. - -It checks experiment configuration and warns about any detected error or inconsistency. -It is used to check if the script is well-formed. -If any template has an inconsistency it will replace them for an empty value on the cmd generated. -Options: -:: - - usage: autosubmit check [-h -nt] expid - - expid experiment identifier - -nt --notransitive - prevents doing the transitive reduction when plotting the workflow - -h, --help show this help message and exit - -Example: -:: - - autosubmit check cxxx - -How to use check in running time: -======================== - -In ``jobs_cxxx.conf`` , you can set check(default true) to check the scripts during autosubmit run cxx. - -There are two parameters related to check: - -* CHECK: Controls the mechanism that allows replacing an unused variable with an empty string ( %_% substitution). It is TRUE by default. - -* SHOW_CHECK_WARNINGS: For debugging purposes. It will print a lot of information regarding variables and substitution if it is set to TRUE. - -.. code-block:: ini - - CHECK = TRUE or FALSE or ON_SUBMISSION # Default is TRUE - SHOW_CHECK_WARNINGS = TRUE or FALSE # Default is FALSE - - - -:: - - CHECK = TRUE # Static templates (existing on `autosubmit create`). Used to substitute empty variables - - CHECK = ON_SUBMISSION # Dynamic templates (generated on running time). Used to substitute empty variables. - - CHECK = FALSE # Used to disable this substitution. - -:: - - SHOW_CHECK_WARNINGS = TRUE # Shows a LOT of information. Disabled by default. - - -For example: - -.. code-block:: ini - - [LOCAL_SETUP] - FILE = filepath_that_exists - PLATFORM = LOCAL - WALLCLOCK = 05:00 - CHECK = TRUE - SHOW_CHECK_WARNINGS = TRUE - ... - [SIM] - FILE = filepath_that_no_exists_until_setup_is_processed - PLATFORM = bsc_es - DEPENDENCIES = LOCAL_SETUP SIM - 1 - RUNNING = chunk - WALLCLOCK = 05:00 - CHECK = ON_SUBMISSION - SHOW_CHECK_WARNINGS = FALSE - ... - - diff --git a/docs/source/usage/configuration/communication_library.rst b/docs/source/usage/configuration/communication_library.rst deleted file mode 100644 index bc2aa4a036e68f4d96d6650e04d7fbe2fc81a757..0000000000000000000000000000000000000000 --- a/docs/source/usage/configuration/communication_library.rst +++ /dev/null @@ -1,19 +0,0 @@ -How to change the communications library -===================================== - -In order to handle the remote communications with the different platforms, Autosubmit uses an implementation -of a communications library. There are multiple implementations, so you can choose any of them. - -.. hint:: - At this moment there are one available communication library which is ``paramiko``. - -To change the communications library, open the /cxxx/conf/autosubmit_cxxx.conf file -where cxxx is the experiment identifier and change the value of the API configuration variable in the communications -section: - -.. code-block:: ini - - [communications] - # Communications library used to connect with platforms: paramiko. - # Default = paramiko - API = paramiko \ No newline at end of file diff --git a/docs/source/usage/configuration/configure.rst b/docs/source/usage/configuration/configure.rst deleted file mode 100644 index c7e7db2006c8d8cc9298ce7dc1d9ebb6d7fbd9d1..0000000000000000000000000000000000000000 --- a/docs/source/usage/configuration/configure.rst +++ /dev/null @@ -1,97 +0,0 @@ -How to configure email notifications -==================================== - -To configure the email notifications, you have to follow two configuration steps: - -1. First you have to enable email notifications and set the accounts where you will receive it. - -Edit ``autosubmit_cxxx.conf`` in the ``conf`` folder of the experiment. - -.. hint:: - Remember that you can define more than one email address divided by a whitespace. - -Example: -:: - - vi /cxxx/conf/autosubmit_cxxx.conf - -.. code-block:: ini - - [mail] - # Enable mail notifications for remote_failures - # Default = True - NOTIFY_ON_REMOTE_FAIL = True - # Enable mail notifications - # Default = False - NOTIFICATIONS = True - # Mail address where notifications will be received - TO = jsmith@example.com rlewis@example.com - -2. Then you have to define for which jobs you want to be notified. - -Edit ``jobs_cxxx.conf`` in the ``conf`` folder of the experiment. - -.. hint:: - You will be notified every time the job changes its status to one of the statuses - defined on the parameter ``NOTIFY_ON`` - -.. hint:: - Remember that you can define more than one job status divided by a whitespace. - -Example: -:: - - vi /cxxx/conf/jobs_cxxx.conf - -.. code-block:: ini - - [LOCAL_SETUP] - FILE = LOCAL_SETUP.sh - PLATFORM = LOCAL - NOTIFY_ON = FAILED COMPLETED - - -How to request exclusivity or reservation -========================================= - -To request exclusivity or reservation for your jobs, you can configure two platform variables: - -Edit ``platforms_cxxx.conf`` in the ``conf`` folder of the experiment. - -.. hint:: - Until now, it is only available for Marenostrum. - -.. hint:: - To define some jobs with exclusivity/reservation and some others without it, you can define - twice a platform, one with this parameters and another one without it. - -Example: -:: - - vi /cxxx/conf/platforms_cxxx.conf - -.. code-block:: ini - - [marenostrum3] - TYPE = LSF - HOST = mn-bsc32 - PROJECT = bsc32 - ADD_PROJECT_TO_HOST = false - USER = bsc32XXX - SCRATCH_DIR = /gpfs/scratch - TEST_SUITE = True - EXCLUSIVITY = True - -Of course, you can configure only one or both. For example, for reservation it would be: - -Example: -:: - - vi /cxxx/conf/platforms_cxxx.conf - -.. code-block:: ini - - [marenostrum3] - TYPE = LSF - ... - RESERVATION = your-reservation-id \ No newline at end of file diff --git a/docs/source/usage/configuration/create_members.rst b/docs/source/usage/configuration/create_members.rst deleted file mode 100644 index c83ea81eead1bb8ecb029057253422b0a8400a26..0000000000000000000000000000000000000000 --- a/docs/source/usage/configuration/create_members.rst +++ /dev/null @@ -1,50 +0,0 @@ -How to create and run only selected members -=========================================== - -Your experiment is defined and correctly configured, but you want to create it only considering some selected members, and also to avoid creating the whole experiment to run only the members you want. Then, you can do it by configuring the setting **RUN_ONLY_MEMBERS** in the file: - -:: - - vi /cxxx/conf/expdef_cxxx.conf - -.. code-block:: ini - - [DEFAULT] - # Experiment identifier - # No need to change - EXPID = cxxx - # HPC name. - # No need to change - HPCARCH = ithaca - - [experiment] - # Supply the list of start dates. Available formats: YYYYMMDD YYYYMMDDhh YYYYMMDDhhmm - # Also you can use an abbreviated syntax for multiple dates with common parts: - # 200001[01 15] <=> 20000101 20000115 - # DATELIST = 19600101 19650101 19700101 - # DATELIST = 1960[0101 0201 0301] - DATELIST = 19900101 - # Supply the list of members. LIST = fc0 fc1 fc2 fc3 fc4 - MEMBERS = fc0 - # Chunk size unit. STRING = hour, day, month, year - CHUNKSIZEUNIT = month - # Chunk size. NUMERIC = 4, 6, 12 - CHUNKSIZE = 1 - # Total number of chunks in experiment. NUMERIC = 30, 15, 10 - NUMCHUNKS = 2 - # Calendar used. LIST: standard, noleap - CALENDAR = standard - # List of members that can be included in this run. Optional. - # RUN_ONLY_MEMBERS = fc0 fc1 fc2 fc3 fc4 - # RUN_ONLY_MEMBERS = fc[0-4] - RUN_ONLY_MEMBERS = - - - -You can set the **RUN_ONLY_MEMBERS** value as shown in the format examples above it. Then, ``Job List`` generation is performed as usual. However, an extra step is performed that will filter the jobs according to **RUN_ONLY_MEMBERS**. It discards jobs belonging to members not considered in the value provided, and also we discard these jobs from the dependency tree (parents and children). The filtered ``Job List`` is returned. - -The necessary changes have been implemented in the API so you can correctly visualize experiments implementing this new setting in **Autosubmit GUI**. - -.. important:: - Wrappers are correctly formed considering the resulting jobs. - diff --git a/docs/source/usage/configuration/index.rst b/docs/source/usage/configuration/index.rst deleted file mode 100644 index 692f0a9fb4b8fb972552842c4e176010ec6566bb..0000000000000000000000000000000000000000 --- a/docs/source/usage/configuration/index.rst +++ /dev/null @@ -1,16 +0,0 @@ -.. _configuration: - -How to configure your experiment -================================= - -.. toctree:: - - configure - new_job - new_platform - communication_library - refresh - create_members - check - update_description - diff --git a/docs/source/usage/configuration/new_job.rst b/docs/source/usage/configuration/new_job.rst deleted file mode 100644 index e8fb396927d2503ca59ab9df21585c663c898750..0000000000000000000000000000000000000000 --- a/docs/source/usage/configuration/new_job.rst +++ /dev/null @@ -1,274 +0,0 @@ -How to add a new job -==================== - -To add a new job, open the /cxxx/conf/jobs_cxxx.conf file where cxxx is the experiment -identifier and add this text:s - -.. code-block:: ini - - [new_job] - FILE = - -This will create a new job named "new_job" that will be executed once at the default platform. This job will user the -template located at (path is relative to project folder). - -This is the minimum job definition and usually is not enough. You usually will need to add some others parameters: - -* PLATFORM: allows you to execute the job in a platform of your choice. It must be defined in the experiment's - platforms.conf file or to have the value 'LOCAL' that always refer to the machine running Autosubmit - -* RUNNING: defines if jobs runs only once or once per start-date, member or chunk. Options are: once, date, - member, chunk - -* DEPENDENCIES: defines dependencies from job as a list of parents jobs separated by spaces. For example, if - 'new_job' has to wait for "old_job" to finish, you must add the line "DEPENDENCIES = old_job". - * For dependencies to jobs running in previous chunks, members or start-dates, use -(DISTANCE). For example, for a job "SIM" waiting for - the previous "SIM" job to finish, you have to add "DEPENDENCIES = SIM-1". - * For dependencies that are not mandatory for the normal workflow behaviour, you must add the char '?' at the end of the dependency. - -* SELECT_CHUNKS (optional): by default, all sections depend on all jobs the items specified on the DEPENDENCIES parameter. However, with this parameter, you could select the chunks of a specific job section. At the end of this doc, you will find diverse examples of this feature. The syntax is as follows: - -.. code-block:: ini - - [jobs] - SELECT_CHUNKS = SIM*[1]*[3] # Enables the dependency of chunk 1 with chunk 3. While chunks 2,4 won't be linked. - SELECT_CHUNKS = SIM*[1:3] # Enables the dependency of chunk 1,2 and 3. While 4 won't be linked. - SELECT_CHUNKS = SIM*[1,3] # Enables the dependency of chunk 1 and 3. While 2 and 4 won't be linked - SELECT_CHUNKS = SIM*[1] # Enables the dependency of chunk 1. While 2, 3 and 4 won't be linked - -* SELECT_MEMBERS (optional): by default, all sections depend on all jobs the items specified on the DEPENDENCIES parameter. However, with this parameter, you could select the members of a specific job section. At the end of this doc, you will find diverse examples of this feature. Caution, you must pick the member index, not the member name. - -.. code-block:: ini - - [expdef.conf] - ... - MEMBERS = AA BB CC DD - ... - [jobs.conf] - SELECT_MEMBERS = SIM*[1]*[3] # Enables the dependency of member BB with member DD. While AA and CC won't be linked. - SELECT_MEMBERS = SIM*[1:3] # Enables the dependency of member BB,CC and DD. While AA won't be linked. - SELECT_MEMBERS = SIM*[1,3] # Enables the dependency of member BB and DD. While AA and CC won't be linked - SELECT_MEMBERS = SIM*[1] # Enables the dependency of member BB. While AA, CC and DD won't be linked - - -* EXCLUDED_CHUNKS (optional): With this parameter, you can prevent the generation of jobs for a list of chunks. - -* EXCLUDED_MEMBERS (optional): With this parameter, you can prevent the generation of jobs for a list of members. - -For jobs running in HPC platforms, usually you have to provide information about processors, wallclock times and more. -To do this use: - -* WALLCLOCK: wallclock time to be submitted to the HPC queue in format HH:MM - -* PROCESSORS: processors number to be submitted to the HPC. If not specified, defaults to 1. - -* THREADS: threads number to be submitted to the HPC. If not specified, defaults to 1. - -* TASKS: tasks number to be submitted to the HPC. If not specified, defaults to 1. - -* HYPERTHREADING: Enables Hyper-threading, this will double the max amount of threads. defaults to false. ( Not available on slurm platforms ) -* QUEUE: queue to add the job to. If not specified, uses PLATFORM default. - -* RETRIALS: Number of retrials if job fails - -* DELAY_RETRY_TIME: Allows to put a delay between retries. Triggered when a job fails. If not specified, Autosubmit will retry the job as soon as possible. Accepted formats are: plain number (there will be a constant delay between retrials, of as many seconds as specified), plus (+) sign followed by a number (the delay will steadily increase by the addition of these number of seconds), or multiplication (*) sign follows by a number (the delay after n retries will be the number multiplied by 10*n). Having this in mind, the ideal scenario is to use +(number) or plain(number) in case that the HPC has little issues or the experiment will run for a little time. Otherwise, is better to use the *(number) approach. - -.. code-block:: ini - - #DELAY_RETRY_TIME = 11 - #DELAY_RETRY_TIME = +11 # will wait 11 + number specified - #DELAY_RETRY_TIME = *11 # will wait 11,110,1110,11110...* by 10 to prevent a too big number - - -There are also other, less used features that you can use: - -* FREQUENCY: specifies that a job has only to be run after X dates, members or chunk. A job will always be created for - the last one. If not specified, defaults to 1 - -* SYNCHRONIZE: specifies that a job with RUNNING=chunk, has to synchronize its dependencies chunks at a 'date' or - 'member' level, which means that the jobs will be unified: one per chunk for all members or dates. - If not specified, the synchronization is for each chunk of all the experiment. - -* RERUN_ONLY: determines if a job is only to be executed in reruns. If not specified, defaults to false. - -* CUSTOM_DIRECTIVES: Custom directives for the HPC resource manager headers of the platform used for that job. - -* SKIPPABLE: When this is true, the job will be able to skip it work if there is an higher chunk or member already ready, running, queuing or in complete status. - -* EXPORT: Allows to run an env script or load some modules before running this job. - -* EXECUTABLE: Allows to wrap a job for be launched with a set of env variables. - -* QUEUE: queue to add the job to. If not specified, uses PLATFORM default. - -Workflow examples: -================== - -Example 1: ----------- - -In this first example, you can see 3 jobs in which last job (POST) shows an example with select chunks: - -.. code-block:: ini - - [INI] - FILE = templates/common/ini.tmpl.sh - RUNNING = member - WALLCLOCK = 00:30 - QUEUE = debug - CHECK = true - - [SIM] - FILE = templates/ecearth3/ecearth3.sim - DEPENDENCIES = INI - RUNNING = chunk - WALLCLOCK = 04:00 - PROCESSORS = 1616 - THREADS = 1 - - [POST] - FILE = templates/common/post.tmpl.sh - DEPENDENCIES = SIM - RUNNING = chunk - WALLCLOCK = 01:00 - QUEUE = Debug - check = true - # Then you can select the specific chunks of dependency SIM with one of those lines: - - SELECT_CHUNKS = SIM*[1]*[3] # Will do the dependency of chunk 1 and chunk 3. While chunks 2,4 won't be linked. - SELECT_CHUNKS = SIM*[1:3] #Enables the dependency of chunk 1,2 and 3. While 4 won't be linked. - SELECT_CHUNKS = SIM*[1,3] #Enables the dependency of chunk 1 and 3. While 2 and 4 won't be linked - SELECT_CHUNKS = SIM*[1] #Enables the dependency of chunk 1. While 2, 3 and 4 won't be linked - -Example 2: select_chunks ------------------------- - -In this workflow you can see an illustrated example of select_chunks used in an actual workflow, to avoid an excess of information we only will see the configuration of a single job: - -.. code-block:: ini - - [SIM] - FILE = templates/sim.tmpl.sh - DEPENDENCIES = INI SIM-1 POST-1 CLEAN-5 - SELECT_CHUNKS = POST*[1] - RUNNING = chunk - WALLCLOCK = 0:30 - PROCESSORS = 768 - -.. figure:: ../../workflows/select_chunks.png - :name: simple - :width: 100% - :align: center - :alt: select_chunks_workflow - -Example 3: SKIPPABLE --------------------- - -In this workflow you can see an illustrated example of SKIPPABLE parameter used in an dummy workflow. - -.. code-block:: ini - - [SIM] - FILE = sim.sh - DEPENDENCIES = INI POST-1 - WALLCLOCK = 00:15 - RUNNING = chunk - QUEUE = debug - SKIPPABLE = TRUE - - [POST] - FILE = post.sh - DEPENDENCIES = SIM - WALLCLOCK = 00:05 - RUNNING = member - #QUEUE = debug - -.. figure:: ../../workflows/skip.png - :name: simple - :width: 100% - :align: center - :alt: skip_workflow - -Example 4: Weak dependencies --------------------- - -In this workflow you can see an illustrated example of weak dependencies. - -Weak dependencies, work like this way: - -* X job only has one parent. X job parent can have "COMPLETED or FAILED" as status for current job to run. -* X job has more than one parent. One of the X job parent must have "COMPLETED" as status while the rest can be "FAILED or COMPLETED". - -.. code-block:: ini - - [GET_FILES] - FILE = templates/fail.sh - RUNNING = chunk - - [IT] - FILE = templates/work.sh - RUNNING = chunk - QUEUE = debug - - [CALC_STATS] - FILE = templates/work.sh - DEPENDENCIES = IT GET_FILES? - RUNNING = chunk - SYNCHRONIZE = member - -.. figure:: ../../workflows/Dashed.png - :name: simple - :width: 100% - :align: center - :alt: dashed_workflow - -Example 5: Select Member --------------------- - -In this workflow you can see an illustrated example of select member. Using 4 members 1 datelist and 4 different job sections. - -Expdef: - -.. code-block:: ini - - [experiment] - DATELIST = 19600101 - MEMBERS = 00 01 02 03 - CHUNKSIZE = 1 - NUMCHUNKS = 2 - -Jobs_conf: - -.. code-block:: ini - - [SIM] - ... - RUNNING = chunk - QUEUE = debug - - [DA] - ... - DEPENDENCIES = SIM - SELECT_MEMBERS = SIM*[0:2] - RUNNING = chunk - SYNCHRONIZE = member - - [REDUCE] - ... - DEPENDENCIES = SIM - SELECT_MEMBERS = SIM*[3] - RUNNING = member - FREQUENCY = 4 - - [REDUCE_AN] - ... - FILE = templates/05b_sim.sh - DEPENDENCIES = DA - RUNNING = chunk - SYNCHRONIZE = member - -.. figure:: ../../workflows/Select_members.png - :name: simple - :width: 100% - :align: center - :alt: select_members \ No newline at end of file diff --git a/docs/source/usage/configuration/new_platform.rst b/docs/source/usage/configuration/new_platform.rst deleted file mode 100644 index 173dafae45e937f818845ae19a69bfa879f841c3..0000000000000000000000000000000000000000 --- a/docs/source/usage/configuration/new_platform.rst +++ /dev/null @@ -1,74 +0,0 @@ -How to add a new platform -========================= - -.. hint:: - If you are interested in changing the communications library, go to the section below. - -To add a new platform, open the /cxxx/conf/platforms_cxxx.conf file where cxxx is the experiment -identifier and add this text: - -.. code-block:: ini - - [new_platform] - TYPE = - HOST = - PROJECT = - USER = - SCRATCH = - - -This will create a platform named "new_platform". The options specified are all mandatory: - -* TYPE: queue type for the platform. Options supported are PBS, SGE, PS, LSF, ecaccess and SLURM. - -* HOST: hostname of the platform - -* PROJECT: project for the machine scheduler - -* USER: user for the machine scheduler - -* SCRATCH_DIR: path to the scratch directory of the machine - -* VERSION: determines de version of the platform type - -.. warning:: With some platform types, Autosubmit may also need the version, forcing you to add the parameter - VERSION. These platforms are PBS (options: 10, 11, 12) and ecaccess (options: pbs, loadleveler). - - -Some platforms may require to run serial jobs in a different queue or platform. To avoid changing the job -configuration, you can specify what platform or queue to use to run serial jobs assigned to this platform: - -* SERIAL_PLATFORM: if specified, Autosubmit will run jobs with only one processor in the specified platform. - -* SERIAL_QUEUE: if specified, Autosubmit will run jobs with only one processor in the specified queue. Autosubmit - will ignore this configuration if SERIAL_PLATFORM is provided - -There are some other parameters that you may need to specify: - -* BUDGET: budget account for the machine scheduler. If omitted, takes the value defined in PROJECT - -* ADD_PROJECT_TO_HOST = option to add project name to host. This is required for some HPCs - -* QUEUE: if given, Autosubmit will add jobs to the given queue instead of platform's default queue - -* TEST_SUITE: if true, autosubmit test command can use this queue as a main queue. Defaults to false - -* MAX_WAITING_JOBS: maximum number of jobs to be waiting in this platform. - -* TOTAL_JOBS: maximum number of jobs to be running at the same time in this platform. - -* CUSTOM_DIRECTIVES: Custom directives for the resource manager of this platform. - -Example: - -.. code-block:: ini - - [platform] - TYPE = SGE - HOST = hostname - PROJECT = my_project - ADD_PROJECT_TO_HOST = true - USER = my_user - SCRATCH_DIR = /scratch - TEST_SUITE = True - CUSTOM_DIRECTIVES = [ "my_directive" ] \ No newline at end of file diff --git a/docs/source/usage/configuration/refresh.rst b/docs/source/usage/configuration/refresh.rst deleted file mode 100644 index 855c853799bd8405bf3534c1de37407792965deb..0000000000000000000000000000000000000000 --- a/docs/source/usage/configuration/refresh.rst +++ /dev/null @@ -1,31 +0,0 @@ -How to refresh the experiment project -===================================== - -To refresh the project directory of the experiment, use the command: -:: - - autosubmit refresh EXPID - -*EXPID* is the experiment identifier. - -It checks experiment configuration and copy code from original repository to project directory. - -.. warning:: DO NOT USE THIS COMMAND IF YOU ARE NOT SURE ! - Project directory ( /proj will be overwritten and you may loose local changes. - - -Options: -:: - - usage: autosubmit refresh [-h] expid - - expid experiment identifier - - -h, --help show this help message and exit - -mc, --model_conf overwrite model conf file - -jc, --jobs_conf overwrite jobs conf file - -Example: -:: - - autosubmit refresh cxxx \ No newline at end of file diff --git a/docs/source/usage/configuration/update_description.rst b/docs/source/usage/configuration/update_description.rst deleted file mode 100644 index 22996238e8b562611abfde6a07ce180544ff91e5..0000000000000000000000000000000000000000 --- a/docs/source/usage/configuration/update_description.rst +++ /dev/null @@ -1,20 +0,0 @@ -.. _updateDescrip: - -How to update the description of your experiment -================================================ - -Use the command: -:: - - autosubmit updatedescrip EXPID DESCRIPTION - -*EXPID* is the experiment identifier. - -*DESCRIPTION* is the new description of your experiment. - -Autosubmit will validate the provided data and print the results in the command line. - -Example: -:: - - autosubmit a29z "Updated using Autosubmit updatedescrip" \ No newline at end of file diff --git a/docs/source/usage/new_experiment/index.rst b/docs/source/usage/new_experiment/index.rst deleted file mode 100644 index 806bd5b7f9f5ef0f835b4e043ac27fdcd825cce8..0000000000000000000000000000000000000000 --- a/docs/source/usage/new_experiment/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -.. _new_experiment: - -How to create your experiment -============================= - -.. toctree:: - - create_exp - test - testcase - diff --git a/docs/source/usage/new_experiment/test.rst b/docs/source/usage/new_experiment/test.rst deleted file mode 100644 index a80333413d9c33d6da148f27cf736b9abd44954f..0000000000000000000000000000000000000000 --- a/docs/source/usage/new_experiment/test.rst +++ /dev/null @@ -1,37 +0,0 @@ -How to test the experiment -========================== -This method is to conduct a test for a given experiment. It creates a new experiment for a given experiment with a -given number of chunks with a random start date and a random member to be run on a random HPC. - -To test the experiment, use the command: -:: - - autosubmit test CHUNKS EXPID - -*EXPID* is the experiment identifier. -*CHUNKS* is the number of chunks to run in the test. - - - -Options: -:: - - usage: autosubmit test [-h] -c CHUNKS [-m MEMBER] [-s STARDATE] [-H HPC] [-b BRANCH] expid - - expid experiment identifier - - -h, --help show this help message and exit - -c CHUNKS, --chunks CHUNKS - chunks to run - -m MEMBER, --member MEMBER - member to run - -s STARDATE, --stardate STARDATE - stardate to run - -H HPC, --HPC HPC HPC to run experiment on it - -b BRANCH, --branch BRANCH - branch from git to run (or revision from subversion) - -Example: -:: - - autosubmit test -c 1 -s 19801101 -m fc0 -H ithaca -b develop cxxx \ No newline at end of file diff --git a/docs/source/usage/new_experiment/testcase.rst b/docs/source/usage/new_experiment/testcase.rst deleted file mode 100644 index dcf8dcfa6a439f00036ebf811ad38280d81da153..0000000000000000000000000000000000000000 --- a/docs/source/usage/new_experiment/testcase.rst +++ /dev/null @@ -1,36 +0,0 @@ -How to create a test case experiment -==================================== -This method is to create a test case experiment. It creates a new experiment for a test case with a -given number of chunks, start date, member and HPC. - -To create a test case experiment, use the command: -:: - - autosubmit testcase - - - -Options: -:: - - usage: autosubmit testcase [-h] [-y COPY] -d DESCRIPTION [-c CHUNKS] - [-m MEMBER] [-s STARDATE] [-H HPC] [-b BRANCH] - - expid experiment identifier - - -h, --help show this help message and exit - -c CHUNKS, --chunks CHUNKS - chunks to run - -m MEMBER, --member MEMBER - member to run - -s STARDATE, --stardate STARDATE - stardate to run - -H HPC, --HPC HPC HPC to run experiment on it - -b BRANCH, --branch BRANCH - branch from git to run (or revision from subversion) - -Example: -:: - - autosubmit testcase -d "TEST CASE cca-intel auto-ecearth3 layer 0: T511L91-ORCA025L75-LIM3 (cold restart) (a092-a09n)" -H cca-intel -b 3.2.0b_develop -y a09n - diff --git a/docs/source/usage/run_modes/index.rst b/docs/source/usage/run_modes/index.rst deleted file mode 100644 index 5284b1efc6647d4f4eeae1baa9b121af256b9ed4..0000000000000000000000000000000000000000 --- a/docs/source/usage/run_modes/index.rst +++ /dev/null @@ -1,16 +0,0 @@ -.. _run_modes: - -How to run your experiment -========================== - -.. toctree:: - - run - run_members - start_time - start_after - rerun - wrappers - remote_dependencies - run_two_step - diff --git a/docs/source/usage/run_modes/remote_dependencies.rst b/docs/source/usage/run_modes/remote_dependencies.rst deleted file mode 100644 index 238480835cfd3d4b1feda1a4779c23bb4428dd8c..0000000000000000000000000000000000000000 --- a/docs/source/usage/run_modes/remote_dependencies.rst +++ /dev/null @@ -1,27 +0,0 @@ -########################################### -Remote Dependencies - Presubmission feature -########################################### - - There is also the possibility of setting the option **PRESUBMISSION** to True in the config directive. - - This allows more than one package containing simple or wrapped jobs to be submitted at the same time, even when the dependencies between jobs aren't yet satisfied. This is only useful for cases when the job scheduler considers the time a job has been queuing to determine the job's priority (and the scheduler understands the dependencies set between the submitted packages). New packages can be created as long as the total number of jobs are below than the number defined in the **TOTALJOBS** variable. - - The jobs that are waiting in the remote platform, will be marked as HOLD. - -How to configure -================ - -In ``autosubmit_cxxx.conf``, regardless of the how your workflow is configured. - -For example: - -.. code-block:: ini - - [config] - EXPID = .... - AUTOSUBMIT_VERSION = 3.13.0b - ... - PRESUBMISSION = TRUE - MAXWAITINGJOBS = 100 - TOTALJOBS = 100 - ... diff --git a/docs/source/usage/run_modes/rerun.rst b/docs/source/usage/run_modes/rerun.rst deleted file mode 100644 index e0410e66c68733af9cdff6ed9c32c12eb924384b..0000000000000000000000000000000000000000 --- a/docs/source/usage/run_modes/rerun.rst +++ /dev/null @@ -1,62 +0,0 @@ -How to rerun a part of the experiment -===================================== - -This procedure allows you to create automatically a new pickle with a list of jobs of the experiment to rerun. - -Using the ``expdef_.conf`` the ``create`` command will generate the rerun if the variable RERUN is set to TRUE and a RERUN_JOBLIST is provided. - -Additionally, you can have re-run only jobs that won't be include in the default job_list. In order to do that, you have to set RERUN_ONLY in the jobs conf of the corresponding job. - -:: - - autosubmit create cxxx - -It will read the list of jobs specified in the RERUN_JOBLIST and will generate a new plot. - -Example: -:: - - vi /cxxx/conf/expdef_cxxx.conf - -.. code-block:: ini - - ... - - [rerun] - RERUN = TRUE - RERUN_JOBLIST = RERUN_TEST_INI;SIM[19600101[C:3]],RERUN_TEST_INI_chunks[19600101[C:3]] - ... - - vi /cxxx/conf/jobs_cxxx.conf - -.. code-block:: ini - - [PREPROCVAR] - FILE = templates/04_preproc_var.sh - RUNNING = chunk - PROCESSORS = 8 - - [RERUN_TEST_INI_chunks] - FILE = templates/05b_sim.sh - RUNNING = chunk - RERUN_ONLY = true - - [RERUN_TEST_INI] - FILE = templates/05b_sim.sh - RUNNING = once - RERUN_ONLY = true - - [SIM] - DEPENDENCIES = RERUN_TEST_INI RERUN_TEST_INI_chunks PREPROCVAR SIM-1 - RUNNING = chunk - PROCESSORS = 10 - - .. figure:: ../../workflows/rerun.png - :name: rerun_result - :align: center - :alt: rerun_result - - -:: - - nohup autosubmit run cxxx & diff --git a/docs/source/usage/run_modes/run.rst b/docs/source/usage/run_modes/run.rst deleted file mode 100644 index a6f242c34ec60b11987e0ef907613c6b66f53150..0000000000000000000000000000000000000000 --- a/docs/source/usage/run_modes/run.rst +++ /dev/null @@ -1,67 +0,0 @@ -How to run the experiment -========================= -Launch Autosubmit with the command: -:: - - autosubmit run EXPID - -*EXPID* is the experiment identifier. - -Options: -:: - - usage: autosubmit run [-h] expid - - expid experiment identifier - -nt --notransitive - prevents doing the transitive reduction when plotting the workflow - -v --update_version - update the experiment version to match the actual autosubmit version - -st --start_time - Sets the starting time for the experiment. Accepted format: 'yyyy-mm-dd HH:MM:SS' or 'HH:MM:SS' (defaults to current day). - -sa --start_after - Sets a experiment expid that will be tracked for completion. When this experiment is completed, the current instance of Autosubmit run will start. - -rm --run_members - Sets a list of members allowed to run. The list must have the format '### ###' where '###' represents the name of the member as set in the conf files. - -h, --help show this help message and exit - -Example: -:: - - autosubmit run cxxx -.. important:: If the autosubmit version is set on autosubmit.conf it must match the actual autosubmit version -.. hint:: It is recommended to launch it in background and with ``nohup`` (continue running although the user who launched the process logs out). - -Example: -:: - - nohup autosubmit run cxxx & - -.. important:: Before launching Autosubmit check password-less ssh is feasible (*HPCName* is the hostname): - -.. important:: The host machine has to be able to access HPC's/Clusters via password-less ssh. Make sure that the ssh key is in PEM format `ssh-keygen -t rsa -b 4096 -C "email@email.com" -m PEM`. - - ``ssh HPCName`` - -More info on password-less ssh can be found at: http://www.linuxproblem.org/art_9.html - -.. caution:: After launching Autosubmit, one must be aware of login expiry limit and policy (if applicable for any HPC) and renew the login access accordingly (by using token/key etc) before expiry. - -How to run an experiment that was created with another version -============================================================== - -.. important:: First of all you have to stop your Autosubmit instance related with the experiment - -Once you've already loaded / installed the Autosubmit version do you want: -:: - - autosubmit create EXPID - autosubmit recovery EXPID -s -all - autosubmit run EXPID -v - or - autosubmit updateversion EXPID - autosubmit run EXPID -v -*EXPID* is the experiment identifier. -The most common problem when you change your Autosubmit version is the apparition of several Python errors. -This is due to how Autosubmit saves internally the data, which can be incompatible between versions. -The steps above represent the process to re-create (1) these internal data structures and to recover (2) the previous status of your experiment. diff --git a/docs/source/usage/run_modes/run_members.rst b/docs/source/usage/run_modes/run_members.rst deleted file mode 100644 index e7aa1eb1bdcfb940560876e0b8c01bf2d1cd7440..0000000000000000000000000000000000000000 --- a/docs/source/usage/run_modes/run_members.rst +++ /dev/null @@ -1,27 +0,0 @@ -How to run only selected members -================================ - -To run only a subset of selected members you can execute the command: -:: - - autosubmit run EXPID -rm MEMBERS - -*EXPID* is the experiment identifier, the experiment you want to run. - -*MEMBERS* is the selected subset of members. Format `"member1 member2 member2"`, example: `"fc0 fc1 fc2"`. - -Then, your experiment will start running jobs belonging to those members only. If the experiment was previously running and autosubmit was stopped when some jobs belonging to other members (not the ones from your input) where running, those jobs will be tracked and finished in the new exclusive run. - -Furthermore, if you wish to run a sequence of only members execution; then, instead of running `autosubmit run -rm "member_1"` ... `autosubmit run -rm "member_n"`, you can make a bash file with that sequence and run the bash file. Example: -:: - - #!/bin/bash - autosubmit run EXPID -rm MEMBER_1 - autosubmit run EXPID -rm MEMBER_2 - autosubmit run EXPID -rm MEMBER_3 - ... - autosubmit run EXPID -rm MEMBER_N - - - - diff --git a/docs/source/usage/run_modes/run_two_step.rst b/docs/source/usage/run_modes/run_two_step.rst deleted file mode 100644 index 4714f165b88e4cf33f3c779c82237c98659672e5..0000000000000000000000000000000000000000 --- a/docs/source/usage/run_modes/run_two_step.rst +++ /dev/null @@ -1,69 +0,0 @@ -############################################################################################## -How to prepare an experiment to run in two independent job_list. (Priority jobs, Two-step-run) -############################################################################################## - -Feature overview ----------------- - -This feature allows to run an experiment in two separated steps without the need of do anything manually. - -To achieve this, you will have to use an special parameter called TWO_STEP_START in which you will put the list of the jobs that you want to run in an exclusive mode. These jobs will run until all of them finishes and once it finishes, the rest of the jobs will begun the execution. - -It can be activated through TWO_STEP_START and it is set on expdef_a02n.conf, under the [experiment] section. - -.. code-block:: ini - - [experiment] - DATELIST = 20120101 20120201 - MEMBERS = fc00[0-3] - CHUNKSIZEUNIT = day - CHUNKSIZE = 1 - NUMCHUNKS = 10 - CHUNKINI = - CALENDAR = standard - # To run before the rest of experiment: - TWO_STEP_START = - -In order to be easier to use, there are Three modes for use this feature: job_names and section,dates,member_or_chunk(M/C),chunk_or_member(C/M). - -* By using job_names alone, you will need to put all jobs names one by one divided by the char , . -* By using section,dates,member_or_chunk(M/C),chunk_or_member(C/M). You will be able to select multiple jobs at once combining these filters. -* Use both options, job_names and section,dates,member_or_chunk(M/C),chunk_or_member(C/M). You will have to put & between the two modes. - -There are 5 fields on TWO_STEP_START, all of them are optional but there are certain limitations: - -* **Job_name**: [Independent] List of job names, separated by ',' char. Optional, doesn't depend on any field. Separated from the rest of fields by '&' must be the first field if specified -* **Section**: [Independent] List of sections, separated by ',' char. Optional, can be used alone. Separated from the rest of fields by ';' -* **Dates**: [Depends on section] List of dates, separated by ',' char. Optional, but depends on Section field. Separated from the rest of fields by ';' -* **member_or_chunk**: [Depends on Dates(OR)] List of chunk or member, must start with C or M to indicate the filter type. Jobs are selected by [1,2,3..] or by a range [0-9] Optional, but depends on Dates field. Separated from the rest of fields by ';' -* **chunk_or_member**: [Depends on Dates(OR)] List of member or chunk, must start with M or C to indicate the filter type. Jobs are selected by [1,2,3..] or by a range [0-9] Optional, but depends on Dates field. Separated from the rest of fields by ';' - -Example -------- - -Guess the expdef configuration as follow: - -.. code-block:: ini - - [experiment] - DATELIST = 20120101 - MEMBERS = 00[0-1] - CHUNKSIZEUNIT = day - CHUNKSIZE = 1 - NUMCHUNKS = 2 - TWO_STEP_START = a02n_20120101_000_1_REDUCE&COMPILE_DA,SIM;20120101;c[1] - -Given this job_list ( jobs_conf has REMOTE_COMPILE(once),DA,SIM,REDUCE) - -['a02n_REMOTE_COMPILE', 'a02n_20120101_000_1_SIM', 'a02n_20120101_000_2_SIM', 'a02n_20120101_001_1_SIM', 'a02n_20120101_001_2_SIM', 'a02n_COMPILE_DA', 'a02n_20120101_1_DA', 'a02n_20120101_2_DA', 'a02n_20120101_000_1_REDUCE', 'a02n_20120101_000_2_REDUCE', 'a02n_20120101_001_1_REDUCE', 'a02n_20120101_001_2_REDUCE'] - -The priority jobs will be ( check TWO_STEP_START from expdef conf): - -['a02n_20120101_000_1_SIM', 'a02n_20120101_001_1_SIM', 'a02n_COMPILE_DA', 'a02n_20120101_000_1_REDUCE'] - - - -Finally, you can launch Autosubmit *run* in background and with ``nohup`` (continue running although the user who launched the process logs out). -:: - - nohup autosubmit run cxxx & diff --git a/docs/source/usage/run_modes/start_after.rst b/docs/source/usage/run_modes/start_after.rst deleted file mode 100644 index 2e6be41813733c1b5e566efd3f0e85e55e1167e4..0000000000000000000000000000000000000000 --- a/docs/source/usage/run_modes/start_after.rst +++ /dev/null @@ -1,20 +0,0 @@ -How to start an experiment after another experiment is finished -=============================================================== - -To start an experiment after another experiment is finished, use the command: -:: - - autosubmit run EXPID -sa EXPIDB - -*EXPID* is the experiment identifier, the experiment you want to start. - -*EXPIDB* is the experiment identifier of the experiment you are waiting for before your experiment starts. - -.. warning:: Both experiments must be using Autosubmit version `3.13.0b` or later. - -Then, your terminal will show the current status of the experiment you are waiting for. The status format is `COMPLETED/QUEUING/RUNNING/SUSPENDED/FAILED`. - -This functionality can be used together with other options supplied by the `run` command. - -The `-sa` command has a long version `--start_after`. - diff --git a/docs/source/usage/run_modes/start_time.rst b/docs/source/usage/run_modes/start_time.rst deleted file mode 100644 index ab83c471c7fe6a1891c429e1319177415979122a..0000000000000000000000000000000000000000 --- a/docs/source/usage/run_modes/start_time.rst +++ /dev/null @@ -1,19 +0,0 @@ -How to start an experiment at a given time -========================================== - -To start an experiment at a given time, use the command: -:: - - autosubmit run EXPID -st INPUT - -*EXPID* is the experiment identifier - -*INPUT* is the time when your experiment will start. You can provide two formats: - * `H:M:S`: For example `15:30:00` will start your experiment at 15:30 in the afternoon of the present day. - * `yyyy-mm-dd H:M:S`: For example `2021-02-15 15:30:00` will start your experiment at 15:30 in the afternoon on February 15th. - -Then, your terminal will show a countdown for your experiment start. - -This functionality can be used together with other options supplied by the `run` command. - -The `-st` command has a long version `--start_time`. diff --git a/docs/source/usage/stats/describe.rst b/docs/source/usage/stats/describe.rst deleted file mode 100644 index 1432108d849a025ee534eb0bc4030e74f046b572..0000000000000000000000000000000000000000 --- a/docs/source/usage/stats/describe.rst +++ /dev/null @@ -1,23 +0,0 @@ -How to get details about the experiment -========================================= -To get details about the experiment, use the command: -:: - - autosubmit describe EXPID - -*EXPID* is the experiment identifier. - -It displays information about the experiment. Currently it describes owner,description_date,model,branch and hpc - -Options: -:: - - usage: autosubmit describe [-h ] expid - - expid experiment identifier - -h, --help show this help message and exit - -Example: -:: - - autosubmit describe cxxx diff --git a/docs/source/usage/stats/index.rst b/docs/source/usage/stats/index.rst deleted file mode 100644 index 41703e13d67a450257131000531897fd53e02dc1..0000000000000000000000000000000000000000 --- a/docs/source/usage/stats/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -.. _stats: - -How to get stats from your experiment -===================================== - -.. toctree:: - - report - stats - describe - diff --git a/docs/source/usage/stats/report.rst b/docs/source/usage/stats/report.rst deleted file mode 100644 index 6f4bb234bddb5e745186a8c0bfcd1c963136489c..0000000000000000000000000000000000000000 --- a/docs/source/usage/stats/report.rst +++ /dev/null @@ -1,87 +0,0 @@ -.. _report: - -How to extract information about the experiment parameters -================================================ - -This procedure allows you to extract the experiment variables that you want. - - -The command can be called with: -:: - - autosubmit report EXPID -t "absolute_file_path" - -Alternatively it also can be called as follows: -:: - - autosubmit report expid -all - -Or combined as follows: -:: - - autosubmit report expid -all -t "absolute_file_path" - -Options: -:: - usage: autosubmit report [-all] [-t] [-fp] [-p] expid - - expid Experiment identifier - - -t, --template Allows to select a set of parameters to be extracted - - -fp, --show_all_parameters All parameters will be extracted to a different file - - -fp, --folder_path By default, all parameters will be put into experiment tmp folder - - -p, --placeholders disable the replacement by - if the variable doesn't exist - -Template format and example: -:: - -Autosubmit parameters are encapsulated by %_%, once you know how the parameter is called you can create a template similar to the one as follows: - -.. code-block:: ini - - - **CHUNKS:** %NUMCHUNKS% - %CHUNKSIZE% %CHUNKSIZEUNIT% - - **VERSION:** %VERSION% - - **MODEL_RES:** %MODEL_RES% - - **PROCS:** %XIO_NUMPROC% / %NEM_NUMPROC% / %IFS_NUMPROC% / %LPJG_NUMPROC% / %TM5_NUMPROC_X% / %TM5_NUMPROC_Y% - - **PRODUCTION_EXP:** %PRODUCTION_EXP% - - **OUTCLASS:** %BSC_OUTCLASS% / %CMIP6_OUTCLASS% - -This will be understood by Autosubmit and the result would be similar to: - -.. code-block:: ini - - - CHUNKS: 2 - 1 month - - VERSION: trunk - - MODEL_RES: LR - - PROCS: 96 / 336 / - / - / 1 / 45 - - PRODUCTION_EXP: FALSE - - OUTCLASS: reduced / - - -Although it depends on the experiment. - -If the parameter doesn't exists, it will be returned as '-' while if the parameter is declared but empty it will remain empty - -List of all parameters example: -:: - -On the other hand, if you use the option -l autosubmit will write a file called parameter_list_.txt containing all parameters in the format as follows: - -.. code-block:: ini - - HPCQUEUE=bsc_es - HPCARCH=marenostrum4 - LOCAL_TEMP_DIR=/home/dbeltran/experiments/ASlogs - NUMCHUNKS=1 - PROJECT_ORIGIN=https://earth.bsc.es/gitlab/es/auto-ecearth3.git - MARENOSTRUM4_HOST=mn1.bsc.es - NORD3_QUEUE=bsc_es - NORD3_ARCH=nord3 - CHUNKSIZEUNIT=month - MARENOSTRUM4_LOGDIR=/gpfs/scratch/bsc32/bsc32070/a01w/LOG_a01w - PROJECT_COMMIT= - SCRATCH_DIR=/gpfs/scratch - HPCPROJ=bsc32 - NORD3_BUDG=bsc32 diff --git a/docs/source/usage/stats/stats.rst b/docs/source/usage/stats/stats.rst deleted file mode 100644 index 6cd368a2fc4ac757e7ab060b5f8047daefa715ad..0000000000000000000000000000000000000000 --- a/docs/source/usage/stats/stats.rst +++ /dev/null @@ -1,106 +0,0 @@ -.. _autoStatistics: - -How to monitor job statistics -============================= -The following command could be adopted to generate the plots for visualizing the jobs statistics of the experiment at any instance: -:: - - autosubmit stats EXPID - -*EXPID* is the experiment identifier. - -Options: -:: - - usage: autosubmit stats [-h] [-ft] [-fp] [-o {pdf,png,ps,svg}] expid - - expid experiment identifier - - -h, --help show this help message and exit - -ft FILTER_TYPE, --filter_type FILTER_TYPE - Select the job type to filter the list of jobs - -fp FILTER_PERIOD, --filter_period FILTER_PERIOD - Select the period of time to filter the jobs - from current time to the past in number of hours back - -o {pdf,png,ps,svg}, --output {pdf,png,ps,svg} - type of output for generated plot - --hide, hide the plot - -nt --notransitive - prevents doing the transitive reduction when plotting the workflow - -Example: -:: - - autosubmit stats cxxx - -The location where user can find the generated plots with date and timestamp can be found below: - -:: - - /cxxx/plot/cxxx_statistics__