SACCTMGR(1)

NAME
sacctmgr - Used to view and modify Slurm account information.

SYNOPSIS
sacctmgr [OPTIONS...] [COMMAND...]

DESCRIPTION
sacctmgr is used to view or modify Slurm account information. The account information is maintained within a database with the interface being provided by slurmdbd (Slurm Database daemon). This database can serve as a central storehouse of user and computer information for multiple computers at a single site. Slurm account information is recorded based upon four parameters that form what is referred to as an association. These parameters are user, cluster, partition, and account. user is the login name. cluster is the name of a Slurm managed cluster as specified by the ClusterName parameter in the slurm.conf configuration file. partition is the name of a Slurm partition on that cluster. account is the bank account for a job. The intended mode of operation is to initiate the sacctmgr command, add, delete, modify, and/or list association records then commit the changes and exit.

OPTIONS

-h, --help
    Print a help message describing the usage of ssacctmgr. This is equivalent to the help command.

-i, --immediate
    commit changes immediately.

-n, --noheader
    No header will be added to the beginning of the output.

-o, --oneliner
    Report output one record per line. This is equivalent to the oneliner command.

-p, --parsable
    Output will be '|' delimited with a '|' at the end.

-P, --parsable2
    Output will be '|' delimited without a '|' at the end.

-Q, --quiet
    Print no messages other than error messages. This is equivalent to the quiet command.

-r, --readonly
    Makes it so the running sacctmgr can not modify accounting information. This is equivalent to the readonly command.

-s, --associations
    Use with show or list to display associations with the entity. This is equivalent to the associations command.

-v, --verbose
    Enable detailed logging. This is equivalent to the verbose command.

-V , --version
    Display version number. This is equivalent to the version command.

COMMANDS

add <ENTITY> <SPECS>
    Add an entity. Identical to the create command.

associations
    Use with show or list to display associations with the entity.

create <ENTITY> <SPECS>
    Add an entity. Identical to the add command.

delete <ENTITY> with <SPECS>
    Delete the specified entities.

dump <ENTITY> with <File=FILENAME>
    Dump cluster data to the specified file.

exit
    Terminate sacctmgr. Identical to the quit command.

help
    Display a description of sacctmgr options and commands.

list <ENTITY> [<SPECS>]
    Display information about the specified entity. By default, all entries are displayed, you can narrow results by specifing SPECS in your query. Identical to the show command.

load <FILENAME>
    Load cluster data to the specified file.

modify <ENTITY> with <SPECS> set <SPECS>
    Modify an entity.

oneliner
    Output one record per line.

quiet
    Print no messages other than error messages.

quit
    Terminate the execution of sacctmgr. Identical to the exit command.

show <ENTITY> [<SPECS>]
    Display information about the specified entity. By default, all entries are displayed, you can narrow results by specifing SPECS in your query. Identical to the list command.

verbose
    Enable detailed logging. This includes time-stamps on data structures, record counts, etc. This is an independent command with no options meant for use in interactive mode.

version
    Display the version number of sacctmgr.

!!  Repeat the last command.

ENTITIES

account
    A bank account, typically specified at job submit time using the --account= option. These may be arranged in a hierarchical fashion, for example accounts chemistry and physics may be children of the account science. The hierarchy may have an arbitrary depth.

association
    The entity used to group information consisting of four parameters: account, cluster, partition (optional), and user.

cluster
    The ClusterName parameter in the slurm.conf configuration file, used to differentiate accounts from on different machines.

configuration
    Used only with the list or show command to report current system configuration.

coordinator
    A special privileged user usually an account manager or such that can add users or sub accounts to the account they are coordinator over. This should be a trusted person since they can change limits on account and user associations inside their realm.

qos Quality of Service (For use with MOAB only).

transaction
    List of transactions that have occurred during a given time period.

user
    The login name.

wckeys
    Workload Characterization Key. An arbitrary string for grouping orthogonal accounts.

GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES

Fairshare=<fairshare>
    Number used in conjunction with other accounts to determine job priority. To clear a previously set value use the modify command with a new value of -1.

GrpCPUMins=<max cpu minutes>
    Maximum number of CPU minutes running jobs are able to be allocated in aggregate for this association and all association which are children of this association. To clear a previously set value use the modify command with a new value of -1. (NOTE: This limit is not enforced if set on the root association of a cluster. So even though it may appear in sacctmgr output it will not be enforced. NOTE: This limit only applys when using the Priority Multifactor plugin. The time is decayed using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf. Currently when this limit is reached jobs will be delayed until they are able to run inside the limit. No jobs will be killed if this limit is reached, this will change in future versions of SLURM.)

GrpCPUs=<max cpus>
    Maximum number of CPUs running jobs are able to be allocated in aggregate for this association and all association which are children of this association. To clear a previously set value use the modify command with a new value of -1. (NOTE: This limit is not currently enforced in SLURM. You can still set this, but have to wait for future versions of SLURM before it is enforced.)

GrpJobs=<max jobs>
    Maximum number of running jobs in aggregate for this association and all association which are children of this association. To clear a previously set value use the modify command with a new value of -1.

GrpNodes=<max nodes>
    Maximum number of nodes running jobs are able to be allocated in aggregate for this association and all association which are children of this association. To clear a previously set value use the modify command with a new value of -1.

GrpSubmitJobs=<max jobs>
    Maximum number of jobs which can be in a pending or running state at any time in aggregate for this association and all association which are children of this association. To clear a previously set value use the modify command with a new value of -1.

GrpWall=<max wall>
    Maximum wall clock time running jobs are able to be allocated in aggregate for this association and all association which are children of this association. To clear a previously set value use the modify command with a new value of -1. (NOTE: This limit is not enforced if set on the root association of a cluster. So even though it may appear in sacctmgr output it will not be enforced. NOTE: This limit only applys when using the Priority Multifactor plugin. The time is decayed using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf. Currently when this limit is reached jobs will be delayed until they are able to run inside the limit. No jobs will be killed if this limit is reached, this will change in future versions of SLURM.)

MaxCPUMins=<max cpu minutes>
    Maximum number of CPU minutes each job is able to use in this account. This is overridden if set directly on a user. Default is the cluster's limit. To clear a previously set value use the modify command with a new value of -1.

MaxCPUs=<max cpus>
    Maximum number of CPUs each job is able to use in this account. This is overridden if set directly on a user. Default is the cluster's limit. To clear a previously set value use the modify command with a new value of -1. (NOTE: This limit is not currently enforced in SLURM. You can still set this, but have to wait for future versions of SLURM before it is enforced.)

MaxJobs=<max jobs>
    Maximum number of jobs each user is allowed to run at one time in this account. This is overridden if set directly on a user. Default is the cluster's limit. To clear a previously set value use the modify command with a new value of -1.

MaxNodes=<max nodes>
    Maximum number of nodes each job is able to use in this account. This is overridden if set directly on a user. Default is the cluster's limit. To clear a previously set value use the modify command with a new value of -1. This is a c-node limit on BlueGene systems.

MaxSubmitJobs=<max jobs>
    Maximum number of jobs which can this account can have in a pending or running state at any time. Default is the cluster's limit. To clear a previously set value use the modify command with a new value of -1.

MaxWall=<max wall>
    Maximum wall clock time each job is able to use in this account. This is overridden if set directly on a user. Default is the cluster's limit. <max wall> format is <min> or <min>:<sec> or <hr>:<min>:<sec> or <days>-<hr>:<min>:<sec> or <days>-<hr>. The value is recorded in minutes with rounding as needed. To clear a previously set value use the modify command with a new value of -1.

QosLevel<operator><comma separated list of qos names>
    (For use with MOAB only.) Specify the default Quality of Service's that jobs are able to run at for this account. To get a list of valid QOS's use 'sacctmgr list qos'. This value will override it's parents value and push down to it's children as the new default. Setting a QosLevel to '' (two single quotes with nothing between them) restores it's default setting. You can also use the operator += and -= to add or remove certain QOS's from a QOS list. Valid <operator> values include:

    =   Set QosLevel to the specified value.

    +=  Add the specified <qos> value to the current QosLevel .

    -=  Remove the specified <qos> value from the current QosLevel.

SPECIFICATIONS FOR ACCOUNTS

Cluster=<cluster>
    Specific cluster to add account to. Default is all in system.

Description=<description>
    An arbitrary string describing an account.

Name=<name>
    The name of a bank account.

Organization=<org>
    Organization to which the account belongs.

Parent=<parent>
    Parent account of this account. Default is the root account, a top level account.

NOTE: If using the WithAssoc option you can also query against association specific information to view only certain associations this account may have. These extra options can be found in the SPECIFICATIONS FOR ASSOCIATIONS section. You can also use the general specifications list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.

LIST/SHOW ACCOUNTS FORMAT OPTIONS

Account
    The name of a bank account.

Description
    An arbitrary string describing an account.

Organization
    Organization to which the account belongs.

Coordinators
    List of users that are a coordinator of the account. (Only filled in when using the WithCoordinator option.)

NOTE: If using the WithAssoc option you can also view the information about the various associations the account may have on all the clusters in the system. The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.

SPECIFICATIONS FOR ASSOCIATIONS

Cluster=<comma separated list of cluster names>
    List the associations of the cluster(s).

Account=<comma separated list of account names>
    List the associations of the account(s).

User=<comma separated list of user names>
    List the associations of the user(s).

Partition=<comma separated list of partition names>
    List the associations of the partition(s).

NOTE: You can also use the general specifications list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section. Other options unique for listing associations:

Tree
    Display account names in a hierarchical fashion.

WithDeleted
    Display information with previously deleted data.

WithSubAccounts
    Display information with subaccounts. Only really valuable when used with the account= option. This will display all the subaccount associations along with the accounts listed in the option.

WOLimits
    Display information without limit information. This is for a smaller default format of Cluster,Account,User,Partition

WOPInfo
    Display information without parent information. (i.e. parent id, and parent account name.) This option also invokes WOPLIMITS.

WOPLimits
    Display information without hierarchical parent limits. (i.e. will only display limits where they are set instead of propagating them from the parent.)

SPECIFICATIONS FOR CLUSTERS

Name=<name>
    The name of a cluster. This should be equal to the ClusterName parameter in the slurm.conf configuration file for some Slurm-managed cluster.

WOLimits
    Display information without limit information. This is for a smaller default format of Cluster,ControlHost,ControlPort,RPC

NOTE: You can also use the general specifications list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.

LIST/SHOW CLUSTER FORMAT OPTIONS

Cluster
    The name of the cluster.

Control Host
    When a slurmctld registers with the database the ip address of the controller is placed here.

Control Port
    When a slurmctld registers with the database the port the controller is listening on is placed here.

CPU Count
    The current count of cpus on the cluster.

NodeNames
    The current Nodes associated with the cluster.

NodeCount
    The current count of nodes associated with the cluster.

RPC When a slurmctld registers with the database the rpc version the controller is running is placed here.

NOTE: You can also view the information about the root association for the cluster. The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.

SPECIFICATIONS FOR COORDINATOR

Accounts=<comma separated list of account names>
    Account name to add this user as a coordinator to.

Names=<comma separated list of user names>
    Names of coordinators.

NOTE: To list coordinators use the WithCoordinator options with list account or list user.

SPECIFICATIONS FOR QOS

Description=<description>
    An arbitrary string describing an account.

Names=<qos>
    Names of qos.

SPECIFICATIONS FOR TRANSACTIONS

Accounts=<comma separated list of account names>
    Only print out the transactions affecting specified accounts.

Action=<Specific action the list will display>

Actor=<Specific name the list will display>

Clusters=<comma separated list of cluster names>
    Only print out the transactions affecting specified clusters.

EndTime=<Date and time where list should end>

StartTime=<Date and time where list should begin>

Users=<comma separated list of user names>
    Only print out the transactions affecting specified users.

WithAssoc
    Get information about which associations were affected by the transactions.

LIST/SHOW TRANSACTIONS FORMAT OPTIONS

Action

Actor

Info

TimeStamp

Where

NOTE: If using the WithAssoc option you can also view the information about the various associations the transaction effected. The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.

SPECIFICATIONS FOR USERS

Account=<account>
    Account name to add this user to.

AdminLevel=<level>
    Admin level of user. Valid levels are None, Operator, and Admin.

Cluster=<cluster>
    Specific cluster to add user to the account on. Default is all in system.

DefaultAccount=<account>
    Identify the default bank account name to be used for a job if none is specified at submission time.

DefaultWCKey=<defaultwckey>
    Identify the default Workload Characterization Key.

Name=<name>
    Name of user.

Partition=<name>
    Partition name.

WCKeys=<wckeys>
    Workload Characterization Key values.

NOTE: If using the WithAssoc option you can also query against association specific information to view only certain associations this account may have. These extra options can be found in the SPECIFICATIONS FOR ASSOCIATIONS section. You can also use the general specifications list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.

LIST/SHOW USER FORMAT OPTIONS

AdminLevel
    Admin level of user.

DefaultAccount
    An the users default account.

Coordinators
    List of users that are a coordinator of the account. (Only filled in when using the WithCoordinator option.)

User
    The name of a user.

NOTE: If using the WithAssoc option you can also view the information about the various associations the user may have on all the clusters in the system. The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.

LIST/SHOW WCKey

WCKey
    Workload Characterization Key.

Cluster
    Specific cluster for the WCKey.

User
    The name of a user for the WCKey.

NOTE: If using the WithAssoc option you can also view the information about the various associations the user may have on all the clusters in the system. The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.

GLOBAL FORMAT OPTION
When using the format option for listing various fields you can put a %NUMBER afterwards to specify how many characters should be printed. i.e. format=name%30 will print 30 characters of field name right justified. A -30 will print 30 characters left justified.

FLAT FILE DUMP AND LOAD
sacctmgr has the capability to load and dump SLURM association data to and from a file. This method can easily add a new cluster or copy an existing clusters associations into a new cluster with similar accounts. Each file contains SLURM association data for a single cluster. Comments can be put into the file with the # character. Each line of information must begin with one of the four titles; Cluster, Parent, Account or User. Following the title is a space, dash, space, entity value, then specifications. Specifications are colon separated. If any variable such as Organization has a space in it surround the name with single or double quotes. To create a file of associations one can run > sacctmgr dump tux file=tux.cfg
(file=tux.cfg is optional) To load a previously created file you can run > sacctmgr load file=tux.cfg Other options for load are - clean - delete what was already there and start from scratch with this information.
Cluster= - specify a different name for the cluster than that which is in the file. Quick explanation how the file works. Since the associations in the system follow a hierarchy so does the file. Any thing that is a parent needs to be defined before any children. The only exception is the understood 'root' account. This is always a default for any cluster and does not need to be defined. To edit/create a file start with a cluster line for the new cluster Cluster - cluster_name:MaxNodesPerJob=15 Anything included on this line will be the defaults for all associations on this cluster. These options are as follows...

GrpCPUMins=
    Maximum number of CPU hours running jobs are able to be allocated in aggregate for this association and all association which are children of this association. (NOTE: This limit is not currently enforced in SLURM. You can still set this, but have to wait for future versions of SLURM before it is enforced.)

GrpCPUs=
    Maximum number of CPUs running jobs are able to be allocated in aggregate for this association and all association which are children of this association. (NOTE: This limit is not currently enforced in SLURM. You can still set this, but have to wait for future versions of SLURM before it is enforced.)

GrpJobs=
    Maximum number of running jobs in aggregate for this association and all association which are children of this association.

GrpNodes=
    Maximum number of nodes running jobs are able to be allocated in aggregate for this association and all association which are children of this association.

GrpSubmitJobs=
    Maximum number of jobs which can be in a pending or running state at any time in aggregate for this association and all association which are children of this association.

GrpWall=
    Maximum wall clock time running jobs are able to be allocated in aggregate for this association and all association which are children of this association.

FairShare=
    To be used with a scheduler like MOAB to determine priority.

MaxJobs=
    Maximum number of jobs the children of this account can run.

MaxNodesPerJob=
    Maximum number of nodes per job the children of this account can run.

MaxProcSecondsPerJob=
    Maximum cpu seconds children of this accounts jobs can run.

MaxWallDurationPerJob=
    Maximum time (not related to job size) children of this accounts jobs can run.

QOS=
    Comma separated list of Quality of Service names (Defined in sacctmgr).

Followed by Accounts you want in this fashion...
    Parent - root (Defined by default)
    Account - cs:MaxNodesPerJob=5:MaxJobs=4:MaxProcSecondsPerJob=20:FairShare=399:MaxWallDurationPerJob=40:Description='Computer Science':Organization='LC'
    Parent - cs
    Account - test:MaxNodesPerJob=1:MaxJobs=1:MaxProcSecondsPerJob=1:FairShare=1:MaxWallDurationPerJob=1:Description='Test Account':Organization='Test'

Any of the options after a ':' can be left out and they can be in any order.
    If you want to add any sub accounts just list the Parent THAT HAS ALREADY BEEN CREATED before the account line in this fashion...

All account options are

Description=
    A brief description of the account.

GrpCPUMins=
    Maximum number of CPU hours running jobs are able to be allocated in aggregate for this association and all association which are children of this association. (NOTE: This limit is not currently enforced in SLURM. You can still set this, but have to wait for future versions of SLURM before it is enforced.)

GrpCPUs=
    Maximum number of CPUs running jobs are able to be allocated in aggregate for this association and all association which are children of this association. (NOTE: This limit is not currently enforced in SLURM. You can still set this, but have to wait for future versions of SLURM before it is enforced.)

GrpJobs=
    Maximum number of running jobs in aggregate for this association and all association which are children of this association.

GrpNodes=
    Maximum number of nodes running jobs are able to be allocated in aggregate for this association and all association which are children of this association.

GrpSubmitJobs=
    Maximum number of jobs which can be in a pending or running state at any time in aggregate for this association and all association which are children of this association.

GrpWall=
    Maximum wall clock time running jobs are able to be allocated in aggregate for this association and all association which are children of this association.

FairShare=
    To be used with a scheduler like MOAB to determine priority.

MaxJobs=
    Maximum number of jobs the children of this account can run.

MaxNodesPerJob=
    Maximum number of nodes per job the children of this account can run.

MaxProcSecondsPerJob=
    Maximum cpu seconds children of this accounts jobs can run.

MaxWallDurationPerJob=
    Maximum time (not related to job size) children of this accounts jobs can run.

Organization=
    Name of organization that owns this account.

QOS(=,+=,-=)
    Comma separated list of Quality of Service names (Defined in sacctmgr).

To add users to a account add a line like this after a Parent - line
    Parent - test
    User - adam:MaxNodesPerJob=2:MaxJobs=3:MaxProcSecondsPerJob=4:FairShare=1:MaxWallDurationPerJob=1:AdminLevel=Operator:Coordinator='test'

All user options are

AdminLevel=
    Type of admin this user is (Administrator, Operator)
    Must be defined on the first occurrence of the user.

Coordinator=
    Comma separated list of accounts this user is coordinator over
    Must be defined on the first occurrence of the user.

DefaultAccount=
    system wide default account name
    Must be defined on the first occurrence of the user.

FairShare=
    To be used with a scheduler like MOAB to determine priority.

MaxJobs=
    Maximum number of jobs this user can run.

MaxNodesPerJob=
    Maximum number of nodes per job this user can run.

MaxProcSecondsPerJob=
    Maximum cpu seconds this user can run per job.

MaxWallDurationPerJob=
    Maximum time (not related to job size) this user can run.

QOS(=,+=,-=)
    Comma separated list of Quality of Service names (Defined in sacctmgr).

ARCHIVE FUNCTIONALITY
Sacctmgr has the capability to archive to a flatfile and or load that data if needed later. The archiving is usually done by the slurmdbd and it is highly recommended you only do it through sacctmgr if you completely understand what you are doing. For slurmdbd options see "man slurmdbd" for more information. Loading data into the database can be done from these files to either view old data or regenerate rolled up data. These are the options for both dump and load of archive information. archive dump

Directory=
    Directory to store the archive data.

Events
    Archive Events. If not specified and PurgeEventMonths is set all event data removed will be lost permanently.

Jobs
    Archive Jobs. If not specified and PurgeJobMonths is set all job data removed will be lost permanently.

PurgeEventMonths=
    Purge cluster event records older than time stated in months.

PurgeJobMonths=
    Purge job records older than time stated in months.

PurgeStepMonths=
    Purge step records older than time stated in months.

PurgeSuspendMonths=
    Purge job suspend records older than time stated in months.

Script=
    Run this script instead of the generic form of archive to flat files.

Steps
    Archive Steps. If not specified and PurgeStepMonths is set all step data removed will be lost permanently.

Suspend
    Archive Suspend Data. If not specified and PurgeSuspendMonths is set all suspend data removed will be lost permanently.

archive load

File=
    File to load into database.

Insert=
    SQL to insert directly into the database. This should be used very cautiously since this is writing your sql into the database.

EXAMPLES
> sacctmgr create cluster tux
> sacctmgr create account name=science fairshare=50
> sacctmgr create account name=chemistry parent=science fairshare=30
> sacctmgr create account name=physics parent=science fairshare=20
> sacctmgr create user name=adam cluster=tux account=physics fairshare=10
> sacctmgr delete user name=adam cluster=tux account=physics
> sacctmgr delete account name=physics cluster=tux
> sacctmgr modify user where name=adam cluster=tux account=physics set
  maxjobs=2 maxwall=30:00
> sacctmgr list associations cluster=tux format=Account,Cluster,User,Fairshare tree withd
> sacctmgr list transactions StartTime=11/03-10:30:00 format=Timestamp,Action,Actor
> sacctmgr dump cluster=tux file=tux_data_file
> sacctmgr load tux_data_file
When modifying an object placing the key words 'set' and the optional 'where' is critical to perform correctly below are examples to produce correct results. As a rule of thumb any thing you put in front of the set will be used as a quantifier. If you want to put a quantifier after the key word 'set' you should use the key word 'where'.
wrong> sacctmgr modify user name=adam set fairshare=10 cluster=tux
This will produce an error as the above line reads modify user adam set fairshare=10 and cluster=tux.
right> sacctmgr modify user name=adam cluster=tux set fairshare=10
right> sacctmgr modify user name=adam set fairshare=10 where cluster=tux
(For use with MOAB only) When changing qos for something only use the '=' operator when wanting to explicitly set the qos to something. In most cases you will want to use the '+=' or '-=' operator to either add to or remove from the existing qos already in place.
If a user already has qos of normal,standby for a parent or it was explicitly set you should use qos+=expedite to add this to the list in this fashion.
> sacctmgr modify user name=adam set qos+=expedite
If you are looking to only add the qos expedite to only a certain account and or cluster you can do that by specifing them in the sacctmgr line.
> sacctmgr modify user name=adam acct=this cluster=tux set qos+=expedite

COPYING
Copyright (C) 2008-2009 Lawrence Livermore National Security. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). CODE-OCEC-09-009. All rights reserved.

This file is part of SLURM, a resource management program. For details, see <https://computing.llnl.gov/linux/slurm/>.

SLURM is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

SLURM is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

SEE ALSO
slurm.conf(5) slurmdbd(8)