From d25fa2fcbb6148c33581f9f8b9d151bab1c6b6c6 Mon Sep 17 00:00:00 2001 From: "Bruno P. Kinoshita" Date: Thu, 10 Aug 2023 16:21:47 +0200 Subject: [PATCH] Use tables, semantic break lines --- docs/source/database/index.rst | 78 +++++++++++++++++++++++++--------- 1 file changed, 58 insertions(+), 20 deletions(-) diff --git a/docs/source/database/index.rst b/docs/source/database/index.rst index 32f3896e7..fa2c22146 100644 --- a/docs/source/database/index.rst +++ b/docs/source/database/index.rst @@ -1,15 +1,28 @@ -######################### +#################### Autosubmit databases -######################### +#################### Introduction ------------ -Autosubmit stores information about its experiments and workflows in SQLite databases and as serialized Python objects (pickle files). These are distributed through the local filesystem, where Autosubmit is installed and runs. +Autosubmit stores information about its experiments and workflows in SQLite +databases and as serialized Python objects (pickle files). These are +distributed through the local filesystem, where Autosubmit is installed and runs. -There is one central database that supports the core functionality of experiments in Autosubmit. There are other auxiliary databases consumed by Autosubmit and the Autosubmit API, that store finer-grained experiment information. +There is one central database that supports the core functionality of +experiments in Autosubmit. There are other auxiliary databases consumed +by Autosubmit and the Autosubmit API, that store finer-grained experiment information. -The name and location of the central database are defined in the .autosubmitrc config file while the other auxiliary DBs have a predefined name. There are also log files with important information about experiment execution and some other relevant information such as experiment job statuses, timestamps, error messages among other things inside these files. +The name and location of the central database are defined in the ``.autosubmitrc`` +configuration file while the other auxiliary databases have a predefined name. +There are also log files with important information about experiment execution and +some other relevant information such as experiment job statuses, timestamps, error +messages among other things inside these files. + +.. note:: + + The ```` is an experiment ID. The location of the databases of + other files can be customized in the ``.autosubmitrc`` configuration file. .. figure:: fig/dbs-highlevel.png :name: simple_database @@ -20,30 +33,55 @@ The name and location of the central database are defined in the .autosubmitrc c Core databases --------------- -* Autosubmit's main database: The default name is autosubmit.db, but the name and location can be customized in ``.autosubmitrc``. Written and read by Autosubmit. -* as_times.db: Used by the Autosubmit API. This database is deprecated since Autosubmit version ``3.x``. It is currently being kept for backward compatibility. Written and read by worker running periodically. ++-------------------+------------------------------------+----------------------------------------------------------------------------------------------------------+ +| Database | Default location | Description | ++===================+====================================+==========================================================================================================+ +| ``autosubmit.db`` | ``$HOME/autosubmit/autosubmit.db`` | The main database of Autosubmit. The location can be customized in the ``autosubmitrc`` file. | ++-------------------+------------------------------------+----------------------------------------------------------------------------------------------------------+ +| ``as_times.db`` | ``$HOME/autosubmit/as_times.db`` | Deprecated API. Used by Autosubmit API with Autosubmit ``3.x``. Kept for backward compatibility for now. | ++-------------------+------------------------------------+----------------------------------------------------------------------------------------------------------+ Auxiliary databases -------------------- -These databases complement the Core database previously described for different purposes, some of them are centralized in the ``$AS_METADATA_FOLDER`` directory (defined in the ``.autosubmitrc`` config file) while others are present inside each experiment folder: -Databases present in ``$AS_METADATA_FOLDER``: +These databases complement the databases previously described for different purposes. +Some of them are centralized in the ``$AS_METADATA`` directory (defined in the +``.autosubmitrc`` config file) while others are present inside each experiment folder. + +Databases in the ``$AS_METADATA`` directory +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -* ``/graph/graph_data_xxxx.db``: used by the GUI to optimize the generation of the graph visualization. Populated by a worker running periodically. -* ``/structures/structure_xxxx.db``: experiment dependencies stored as an edge list. Used in the GUI and populated by a worker running periodically. -* ``/data/job_data_xxxx.db``: Stores incremental historical job data information for a given experiment and also some other metrics, filled by Autosubmit during the job handling, there is one per experiment. -* ``/test/status.db``: Stores status of the partition where all Autosubmit DBs and experiment files are stored, populated by a worker running periodically. ++---------------------------+---------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| Database | Default location | Description | ++===========================+===============================================================+============================================================================================================================+ +| ``graph_data_.db`` | ``$HOME/autosubmit/metadata/graph/graph_data_.db`` | Used by the GUI to improve the graph visualization. Populated by an API worker. | ++---------------------------+---------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| ``structure_.db`` | ``$HOME/autosubmit/metadata/structures/structure_.db`` | Used by the GUI to display edge lists. Populated by an API worker. | ++---------------------------+---------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| ``status.db`` | ``$HOME/autosubmit/metadata/test/status.db`` | Stores the status of the partition where Autosubmit databases and experiment files are stored. Populated by an API worker. | ++---------------------------+---------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| ``job_data_.db`` | ``$HOME/autosubmit/metadata/data/job_data_.db`` | Stores experiment metrics and historical information. Populated by Autosubmit. | ++---------------------------+---------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ -Note that ``xxxx`` is the ID of a given experiment, and also that the root path (``$AS_METADATA_FOLDER``) is determined by the configuration defined in ``.autosubmitrc`` config file, under the path defined there, the folder mentioned above will be created. -Databases in each experiment folder: +Databases in each experiment directory +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -* ``job_packages_xxxx.db``: Stores the wrappers defined in the experiment, if no wrapper is defined then it may not exist or be empty. -* ``structure_xxxx.db``: Now deprecated, present in older experiments, same structure and purpose as the one described in the previous section. ++-----------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------+ +| Database | Default location | Description | ++=============================+==========================================================+====================================================================================================================+ +| ``job_packages_.db`` | ``$HOME/autosubmit//pkl/job_packages_.db`` | Stores information about the wrappers configured in the experiment. Empty if no wrappers configured. | ++-----------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------+ +| ``structure_.db`` | ``$HOME/autosubmit//pkl/structure_.db`` | Deprecated. Used in Autosubmit ``3.x``, now replaced by the database used by the Autosubmit API (described above). | ++-----------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------+ Other files ----------- -Python Pickle files (.pkl): it has the defined job list of the experiment with state information of all of them. So in the event of a crash or the user stops the experiment, Autosubmit can resume from the last valid state stored in this file. -Update lists: to change the status of experiment jobs without stopping Autosubmit, it is a text file. +Autosubmit stores Pickle files (e.g. ``$HOME/autosubmit//pkl/job_list_.pkl``) +with the job list of experiments. In the event of a crash, or if the user stops the +experiment, that Pickle file is used in order to be able to restore the experiment to +its latest status. -These files are present in the experiment folder. \ No newline at end of file +There are also update list files, used to change the status of experiment jobs +without stopping Autosubmit. These files are plain text files, and also present +in the experiment directory. -- GitLab