# Accounting

SLURM collects accounting information for every job and job step executed. Information is available about both currently executing jobs and jobs which have already terminated and can be viewed using the sacct command. Resource usage is reported for each task and this can be useful to detect load imbalance between the tasks. SLURM version 1.2 and earlier supported the storage of accounting records to a text file. Beginning in SLURM version 1.3 accounting records can be written to a database.

There are three distinct plugin types associated with resource accounting. The configuration parameters associated with these plugins include:

- JobCompType controls how job completion information is recorded. This can be used to record basic job information such as job name, user name, allocated nodes, start time, completion time, exit status, etc. If the preservation of only basic job information is required, this plugin should satisfy your needs with minimal overhead. You can store this information in a text file, MySQL or PostgreSQL database optionally using either Gold or SlurmDBD for added database security.
- JobAcctGatherType is operating system dependent and controls what mechanisms are used to collect accounting information. Supported values are jobacct_gather/aix, jobacct_gather/linux and jobacct_gather/none (no information collected).
- AccountingStorageType controls how detailed job and job step information is recorded. You can store this information in a text file, MySQL or PostgreSQL database optionally using either Gold or SlurmDBD for added security.

Storing the information into text files is very simple. Just configure the appropriate plugin (e.g. AccountingStorageType=accounting_storage/filetxt and/or JobCompType=jobcomp/filetxt) and then specify the pathname of the file (e.g. AccountingStorageLoc=/var/log/slurm/accounting and/or JobCompLoc=/var/log/slurm/job_completions). Use the logrotate or similar tool to prevent the log files from getting too large. Send a SIGHUP signal to the slurmctld deaemon after moving the files, but before compressing them so that new log files will be created.

Storing the data directly into a database from SLURM may seem attractive, but that requires the availability of user name and password data not only for the SLURM control daemon (slurmctld), but also user commands which need to access the data (sacct and sacctmgr). Making information available to all users makes database security more difficult to provide, sending the data through an intermediate daemon can provide better security. Gold and SlurmDBD are two such services. Our initial implementation relied upon Gold, but we found its performance to be inadequate for our needs and developed SlurmDBD. SlurmDBD (SLURM Database Daemon) is written in C, multi-threaded, secure, and considerably faster than Gold. The configuration required to use SlurmDBD will be described below. Direct database or Gold use would be similar.

Note that SlurmDBD relies upon existing SLURM plugins for authentication and database use, but the other SLURM commands and daemons are not required on the host where SlurmDBD is installed. Install the slurmdbd and slurm-plugins RPMs on the computer when SlurmDBD is to execute.

## Infrastructure

If the SlurmDBD is executed on a different cluster than the one managed by SLURM, possibly to collect data from multiple clusters in a single location, there are some constraints on the user space. The user ID associated with SlurmUser must be uniform across all clusters. Accounting is maintained by user name (not user ID), but a given user name should refer to the same person across all of the computers.

The best way to insure security of the data is by authenticating communications to the SlurmDBD and we recommend Munge for that purpose. Munge was designed to support authentication within a cluster. If you have one cluster managed by SLURM and execute the SlurmDBD on that one cluster, the normal Munge configuration will suffice. Otherwise Munge should then be installed on all nodes of all SLURM managed clusters plus the machine where SlurmDBD executes. You then have a choice of either having a single Munge key for all of these computers or maintaining a unique key for each of the clusters plus a second key for communications between the clusters for better security. Munge enhancements are planned to support two keys within a single configuration file, but presently two different daemons must be started with different configuration files to support two different keys. If a Munge separate daemon configured to provide enterprise-wide authentication, it will have a unique named pipe configured for communications. The pathname of this pipe will be needed in the SLURM and SlurmDBD configuration files (slurm.conf and slurmdbd.conf respectively, more details are provided below).

## SLURM Configuration

Several SLURM configuration parameters must be set to support archiving information in SlurmDBD. SlurmDBD has a separate configuration file which is documented in a separate section. Note that you can write accounting information to SlurmDBD while job completion records are written to a text file or not maintained at all. If you don't set the configuration parameters that begin with "JobComp" then job completion records will not be recorded.

- AccountingStorageEnforce: If you want to prevent users from running jobs if their association (a combination of cluster, account, and user names. For more flexibility in accounting the association can also include a partition name, but it is not necissary.) is not in the database, then set this to "1". Otherwise jobs will be executed based upon policies configured in SLURM on each cluster.
- AccountingStorageHost: The name or address of the host where SlurmDBD executes
- AccountingStoragePass: If using SlurmDBD with a second Munge daemon, store the pathname of the named socket used by Munge to provide enterprise-wide. Otherwise the default Munge daemon will be used. .
- AccountingStoragePort: The network port that SlurmDBD accepts communication on.
- AccountingStorageType: Set to "accounting_storage/slurmdbd".
- ClusterName: Set to a unique name for each Slurm-managed cluster so that accounting records from each can be identified.
- JobCompHost: The name or address of the host where SlurmDBD executes.
- JobCompPass: If using SlurmDBD with a second Munge daemon, store the pathname of the named socket used by Munge to provide enterprise-wide. Otherwise the default Munge daemon will be used.
- JobCompPort: The network port that SlurmDBD accepts communication on.
- JobCompType: Set to "jobcomp/slurmdbd".

## SlurmDBD Configuration

SlurmDBD requires its own configuration file called "slurmdbd.conf". This file should be only on the computer where SlurmDBD executes and should only be readable by the user which executes SlurmDBD (e.g. "slurm"). This file should be protected from unauthorized access since it contains a database login name and password. See "man slurmdbd.conf" for a more complete description of the configuration parameters. Some of the more important parameters include:

- AuthInfo: If using SlurmDBD with a second Munge daemon, store the pathname of the named socket used by Munge to provide enterprise-wide. Otherwise the default Munge daemon will be used.
- AuthType: Define the authentication method for communications between SLURM components. A value of "auth/munge" is recommended.
- DbdHost: The name of the machine where the Slurm Database Daemon is executed. This should be a node name without the full domain name (e.g. "lx0001"). This value must be specified.
- DbdPort: The port number that the Slurm Database Daemon (slurmdbd) listens to for work. The default value is SLURMDBD_PORT as established at system build time. If none is explicitly specified, it will be set to 6819. This value must be equal to the SlurmDbdPort parameter in the slurm.conf file.
- LogFile: Fully qualified pathname of a file into which the Slurm Database Daemon's logs are written. The default value is none (performs logging via syslog).
- PluginDir: Identifies the places in which to look for SLURM plugins. This is a colon-separated list of directories, like the PATH environment variable. The default value is "/usr/local/lib/slurm".
- SlurmUser: The name of the user that the slurmctld daemon executes as. This user must exist on the machine executing the Slurm Database Daemon and have the same user ID as the hosts on which slurmctld execute. For security purposes, a user other than "root" is recommended. The default value is "root".
- StorageHost: Define the name of the host the database is running where we are going to store the data. Ideally this should be the host on which slurmdbd executes.
- StorageLoc: Specifies the name of the database where accounting records are written, for databases the default database is slurm_acct_db. Note the name can not have a '/' in it or the default will be used.
- StoragePass: Define the password used to gain access to the database to store the job accounting data.
- StoragePort: Define the port on which the database is listening.
- StorageType: Define the accounting storage mechanism type. Acceptable values at present include "accounting_storage/gold", "accounting_storage/mysql", and "accounting_storage/pgsql". The value "accounting_storage/gold" indicates that account records will be written to Gold, which maintains its own database. Use of Gold is not recommended due to reduced performance without providing any additional security. The value "accounting_storage/mysql" indicates that accounting records should be written to a MySQL database specified by the StorageLoc parameter. The value "accounting_storage/pgsql" indicates that accounting records should be written to a PostgreSQL database specified by the StorageLoc parameter. This value must be specified.
- StorageUser: Define the name of the user we are going to connect to the database with to store the job accounting data.

## Tools

There are two tools available to work with accounting data, sacct and sacctmgr. Both of these tools will get or set data through the SlurmDBD daemon. Sacct is used to generate accounting report for both running and completed jobs. Sacctmgr is used to manage associations in the database: add or remove clusters, add or remove users, etc. See the man pages for each command for more information.

Web interfaces with graphical output is currently under development and should be available in the summer of 2008. A tool to report node state information is also under development.

## Database Configuration

Accounting records are maintained based upon what we refer to as an Association, which consists of four elements: cluster, account, and user names. For more flexibility in accounting the association can also include a partition name, but it is not necessary. Use the sacctmgr command to create and manage these records. There is an order to set up accounting associations. You must define clusters before you add accounts and you must add accounts before you can add users.

For example, to add a cluster named "snowflake" to the database execute this line:

    sacctmgr add cluster snowflake

Add accounts "none" and "test" to cluster "snowflake" with an execute line of this sort:

    sacctmgr add account none,test Cluster=snowflake \
      Description="none" Organization="none"

If you have more clusters you want to add these accounts to you can either not specify a cluster, which will add the accounts to all clusters in the system, or comma separate the cluster names you want to add to in the cluster option. Note that multiple accounts can be added at the same time by comma separating the names. Some Description of the account and the organization which it belongs must be specified. These terms can be used later to generated accounting reports. Accounts may be arranged in a hierarchical fashion, for example accounts chemistry and physics may be children of the account science. The hierarchy may have an arbitrary depth. To do this one only needs to specify the parent='' option in the add account line. For the example above execute

    sacctmgr add account science \
       Description="science accounts" Organization=science
    sacctmgr add account chemistry,physics parent=science \
       Description="physical sciences" Organization=science

Add users to accounts using similar syntax. For example, to permit user da to execute jobs on all clusters with a default account of test execute:

    sacctmgr add user da default=test

If AccountingStorageEnforce=1 is configured in the slurm.conf of the cluster snowflake then user da would be allowed to run in account test and any other accounts added in the future. Any attempt to use other accounts will result in the job being aborted. Account test will be the default if he doesn't specify one in a srun line.

Partition names can also be added to an "add user" command with the Partition='partitionname' option to specify an association specific to a slurm partition.

## Account Options

When either adding or modifying an account, the following sacctmgr options are available:

- Description= Description of the account. (Required on creation)
- Organization=Organization of the account. (Required on creation)
- Name= Name of account
- Cluster= Only add this account to these clusters. The account is added to all defined clusters by default.
- Parent= Make this account a child of this other account.

## User Options

When either adding or modifying a user, the following sacctmgr options are available:

- Name= User name
- DefaultAccount= Default account for the user, used when no account is specified when a job is sumbitted. (Required on creation)
- AdminLevel= This field is used to allow a user to add accounting privileges to this user. Valid options are
  - None
  - Operator: can add, modify,and remove users, and add other operators)
  - Admin: In addition to operator privileges these users can add, modify, and remove accounts and clusters
- Account= Account(s) to add user to
- Cluster= Only add to accounts on these clusters (default is all clusters)
- Partition= Name of Slurm partition this association applies to

## Modifying Entities

When modifying entities, you can specify many different options in SQL-like fashion, using key words like where and set. A typical execute line has the following form:

    sacctmgr modify <entity> set <options> where <options>

For example:

    sacctmgr modify user set default=none where default=test

will change all users with a default account of "test" to account "none". Once an entity has been added, modified or removed, the change is sent to the appropriate SLURM daemons and will be available for use instantly.

## Removing Entities

Removing entities using an execute line similar to the modify example above, but without the set options. For example, remove all users with a default account "test" using the following execute line:

    sacctmgr remove user where default=test

## Node State Information

Node state information is also recorded in the database. Whenever a node goes DOWN or becomes DRAINED that event is logged along with the node's Reason field. This can be used to generate various reports.

Last modified 25 March 2008