slurm.conf(5)

NAME
slurm.conf - Slurm configuration file

DESCRIPTION
/etc/slurm.conf is an ASCI file which describes general SLURM configuration information, the nodes to be managed, information about how those nodes are grouped into partitions, and various scheduling parameters associated with those partitions. The file location can be modified at system build time using the DEFAULT_SLURM_CONF parameter.

The contents of the file are case insensitive except for the names of nodes and partitions. Any text following a "#" in the configuration file is treated as a comment through the end of that line. The size of each line in the file is limited to 1024 characters. Changes to the configuration file take effect upon restart of SLURM daemons, daemon receipt of the SIGHUP signal, or execution of the command "scontrol reconfigure" unless otherwise noted.

The overall configuration parameters available include:

AuthType
    Define the authentication method for communications between SLURM components. Acceptable values at present include "auth/none", "auth/authd", and "auth/munge". The default value is "auth/none", which means the UID included in communication messages is not verified. This may be fine for testing purposes, but do not use "auth/none" if you desire any security. "auth/authd" indicates that Brett Chun's authd is to be used (see "http://www.theether.org/authd/" for more information). "auth/munge" indicates that Chris Dunlap's munge is to be used (this is the best supported authentication mechanism for SLURM, see "http://www.llnl.gov/linux/munge/" for more information). All SLURM daemons and commands must be terminated prior to changing the value of AuthType and later restarted (SLURM jobs can be preserved).

BackupAddr
    Name that BackupController should be referred to in establishing a communications path. This name will be used as an argument to the gethostbyname() function for identification. For example, "elx0000" might be used to designate the ethernet address for node "lx0000". By default the BackupAddr will be identical in value to BackupController.

BackupController
    The name of the machine where SLURM control functions are to be executed in the event that ControlMachine fails. This node may also be used as a compute server if so desired. It will come into service as a controller only upon the failure of ControlMachine and will revert to a "standby" mode when the ControlMachine becomes available once again. This should be a node name without the full domain name (e.g. "lx0002"). While not essential, it is recommended that you specify a backup controller.

ControlAddr
    Name that ControlMachine should be referred to in establishing a communications path. This name will be used as an argument to the gethostbyname() function for identification. For example, "elx0000" might be used to designate the ethernet address for node "lx0000". By default the ControlAddr will be identical in value to ControlMachine.

ControlMachine
    The name of the machine where SLURM control functions are executed. This should be a node name without the full domain name (e.g. "lx0001"). This value must be specified.

Epilog
    Fully qualified pathname of a script to execute as user root on every node when a user's job completes (e.g. "/usr/local/slurm/epilog"). This may be used to purge files, disable user login, etc. By default there is no epilog.

FastSchedule
    If set to 1 (the default), then consider the configuration of each node to be that specified in the configuration file. If set to 0, then base scheduling decisions upon the actual configuration of each individual node. If the number of node configuration entries in the configuration file is significantly lower than the number of nodes, setting FastSchedule to 1 will permit much faster scheduling decisions to be made. (The scheduler can just check the values in a few configuration records instead of possibly thousands of node records. If a job can't be initiated immediately, the scheduler may execute these tests repeatedly.)

FirstJobId
    The job id to be used for the first submitted to SLURM without a specific requested value. Job id values generated will incremented by 1 for each subsequent job. This may be used to provide a meta-scheduler with a job id space which is disjoint from the interactive jobs. The default value is 1.

HashBase
    If the node names include a sequence number, this value defines the base to be used in building a hash table based upon node name. Value of 8 and 10 are recognized for octal and decimal sequence numbers respectively. The value of zero is also recognized for node names lacking a sequence number. The use of node names containing a numeric suffix will provide faster operation for larger clusters. The default value is 10.

HeartbeatInterval
    The interval, in seconds, at which the SLURM controller tests the status of other daemons. The default value is 30 seconds.

InactiveLimit
    The interval, in seconds, a job or job step is permitted to be inactive before it is terminated. A job or job step is considered inactive if the associated srun command is not responding to slurm daemons. This could be due to the termination of the srun command or the program being is a stopped state. A batch job is considered inactive if it has no active job steps (e.g. periods of pre- and post-processing). This limit permits defunct jobs to be purged in a timely fashion without waiting for their time limit to be reached. This value should reflect the possibility that the srun command may stopped by a debugger or considerable time could be required for batch job pre- and post-processing. The default value is unlimited (zero).

JobCompLoc
    Define the location where job completion records are to be logged. The interpretation of this value depends upon the logging mechanism specified by the JobCompType parameter.

JobCompType
    Define the job completion logging mechanism type. Acceptable values at present include "jobcomp/none" and "jobcomp/filetxt". The default value is "jobcomp/none", which means that upon job completion the record of the job is purged from the system. The value "jobcomp/filetxt" indicates that a record of the job should be written to a text file specified by the JobCompLoc parameter.

JobCredentialPrivateKey
    Fully qualified pathname of a file containing a private key used for authentication by Slurm daemons.

JobCredentialPublicCertificate
    Fully qualified pathname of a file containing a public key used for authentication by Slurm daemons.

KillWait
    The interval, in seconds, given to a job's processes between the SIGTERM and SIGKILL signals upon reaching its time limit. If the job fails to terminate gracefully in the interval specified, it will be forcably terminated. The default value is 30 seconds.

MaxJobCount
    The maximum number of jobs SLURM can have in its active database at one time. Set the values of MaxJobCount and MinJobAge to insure the slurmctld daemon does not exhaust its memory or other resources. Once this limit is reached, requests to submit additional jobs will fail. The default value is 2000 jobs. This value may not be reset via "scontrol reconfig". It only takes effect upon restart of the slurmctld daemon.

MinJobAge
    The minimum age of a completed job before its record is purged from SLURM's active database. Set the values of MaxJobCount and MinJobAge to insure the slurmctld daemon does not exhaust its memory or other resources. The default value is 300 seconds. A value of zero prevents any job record purging.

PluginDir
    Identifies the places in which to look for SLURM plugins. This is a colon-separated list of directories, like the PATH environment variable. The default value is "/usr/local/lib/slurm".

Prolog
    Fully qualified pathname of a script to execute as user root on every node when a user's job begins execution (e.g. "/usr/local/slurm/prolog"). This may be used to purge files, enable user login, etc. By default there is no prolog.

ReturnToService
    If set to 1, then a DOWN node will become available for use upon registration. The default value is 0, which means that a node will remain in the DOWN state until a system administrator explicitly changes its state (even if the slurmd daemon registers and resumes communications).

SchedulerAuth
    An authentication token, if any, that must be used in a scheduler communication protocol. The interpretation of this value depends upon the value of SchedulerType. In the Wiki scheduler plugin, this value must correspond to the checksum seed with which Maui was compiled.

SchedulerPort
    The port number on which slurmctld should listen for connection requests. This value is only used by the Maui Scheduler (see SchedulerType).

SchedulerType
    Identifies the type of scheduler to be used. Acceptable values include "sched/builtin" for the built-in FIFO scheduler, "sched/backfill" for a backfill scheduling module to augment the default FIFO scheduling, and "sched/wiki" for the Wiki interface to the Maui Scheduler. The default value is "sched/builtin". When initially setting the value to "sched/wiki", any pending jobs must have their priority set to zero (held). When changing the value from "sched/wiki", all pending jobs should have their priority change from zero to some large number. The scontrol command can be used to change job priorities. The slurmctld daemon must be restarted for a change in scheduler type to become effective.

SlurmUser
    The name of the user that the slurmctld daemon executes as. For security purposes, a user other than "root" is recommended. The default value is "root".

SlurmctldDebug
    The level of detail to provide slurmctld daemon's logs. Values from 0 to 7 are legal, with `0' being "quiet" operation and `7' being insanely verbose. The default value is 3.

SlurmctldLogFile
    Fully qualified pathname of a file into which the slurmctld daemon's logs are written. The default value is none (performs logging via syslog).

SlurmctldPidFile
    Fully qualified pathname of a file into which the slurmctld daemon may write its process id. This may be used for automated signal processing. The default value is "/var/run/slurmctld.pid".

SlurmctldPort
    The port number that the SLURM controller, slurmctld, listens to for work. The default value is SLURMCTLD_PORT as established at system build time. NOTE: Either slurmctld and slurmd daemons must not execute on the same nodes or the values of SlurmctldPort and SlurmdPort must be different.

SlurmctldTimeout
    The interval, in seconds, that the backup controller waits for the primary controller to respond before assuming control. The default value is 120 seconds.

SlurmdDebug
    The level of detail to provide slurmd daemon's logs. Values from 0 to 7 are legal, with `0' being "quiet" operation and `7' being insanely verbose. The default value is 3.

SlurmdLogFile
    Fully qualified pathname of a file into which the slurmd daemon's logs are written. The default value is none (performs logging via syslog).

SlurmdPidFile
    Fully qualified pathname of a file into which the slurmd daemon may write its process id. This may be used for automated signal processing. The default value is "/var/run/slurmd.pid".

SlurmdPort
    The port number that the SLURM compute node daemon, slurmd, listens to for work. The default value is SLURMD_PORT as established at system build time. NOTE: Either slurmctld and slurmd daemons must not execute on the same nodes or the values of SlurmctldPort and SlurmdPort must be different.

SlurmdSpoolDir
    Fully qualified pathname of a directory into which the slurmd daemon's state information and batch job script information are written. This must be a common pathname for all nodes, but should represent a directory which is local to each node (reference a local file system). The default value is "/var/spool/slurmd." NOTE: This directory is also used to store slurmd's shared memory lockfile, and should not be changed unless the system is being cleanly restarted. If the location of SlurmdSpoolDir is changed and slurmd is restarted, the new daemon will attach to a different shared memory region and lose track of any running jobs.

SlurmdTimeout
    The interval, in seconds, that the SLURM controller waits for slurmd to respond before configuring that node's state to DOWN. The default value is 300 seconds. A value of zero indicates the node should never be set DOWN if not responding.

StateSaveLocation
    Fully qualified pathname of a directory into which the SLURM controller, slurmctld, saves its state (e.g. "/usr/local/slurm/checkpoint"). SLURM state will saved here to recover from system failures. SlurmUser must be able to create files in this directory. The default value is "/tmp". If any slurm daemons terminate abnormally, their core files will also be written into this directory.

SwitchType
    Identifies the type of switch or interconnect used for application communications. Acceptable values include "switch/none" for switches not requiring special processing for job launch or termination (Myrinet, Ethernet, and InfiniBand), "switch/elan" for Quadrics Elan 3 or Elan 4 interconnect. The default value is "switch/none". All SLURM daemons, commands and running jobs must be restarted for a change in SwitchType to take effect. If running jobs exist at the time slurmctld is restarted with a new value of SwitchType, records of all jobs in any state may be lost.

TmpFS
    Fully qualified pathname of the file system available to user jobs for temporary storage. This parameter is used in establishing a node's TmpDisk space. The default value is "/tmp".

WaitTime
    Specifies how many seconds the srun command should by default wait after the first task terminates before terminating all remaining tasks. The "--wait" option on the srun command line overrides this value. If set to 0, this feature is disabled.

The configuration of nodes (or machines) to be managed by Slurm is also specified in /etc/slurm.conf. Only the NodeName must be supplied in the configuration file. All other node configuration information is optional. It is advisable to establish baseline node configurations, especially if the cluster is heterogeneous. Nodes which register to the system with less than the configured resources (e.g. too little memory), will be placed in the "DOWN" state to avoid scheduling jobs on them. Establishing baseline configurations will also speed SLURM's scheduling process by permitting it to compare job requirements against these (relatively few) configuration parameters and possibly avoid having to check job requirements against every individual node's configuration. The resources checked at node registration time are: Procs, RealMemory and TmpDisk. While baseline values for each of these can be established in the configuration file, the actual values upon node registration are recorded and these actual values may be used for scheduling purposes (depending upon the value of FastSchedule in the configuration file.

Default values can be specified with a record in which "NodeName" is "DEFAULT". The default entry values will apply only to lines following it in the configuration file and the default values can be reset multiple times in the configuration file with multiple entries where "NodeName=DEFAULT". The "NodeName=" specification must be placed on every line describing the configuration of nodes. In fact, it is generally possible and desirable to define the configurations of all nodes in only a few lines. This convention permits significant optimization in the scheduling of larger clusters. In order to support the concept of jobs requiring consecutive nodes on some architectures, node specifications should be place in this file in consecutive order. If a specific node name is listed more than once in the configuration file only its "State" and "Reason" fields may be reset. This may be useful to record the state of nodes which are temporarily in a DOWN or DRAINED state without altering permanent configuration information as shown in the example. The node configuration specifies the following information:

NodeName
    Name of a node as returned by the hostname command, without the full domain name (e.g. "lx0012"). A simple node range expression may optionally be used to specify ranges of nodes to avoid building a configuration file with large numbers of entries. The node range expression can contain one pair of square brackets with a sequence of comma separated numbers and/or ranges of numbers separated by a "-" (e.g. "linux[0-64,128]", or "lx[15,18,32-33]"). If the NodeName is "DEFAULT", the values specified with that record will apply to subsequent node specifications unless explicitly set to other values in that node record or replaced with a different set of default values. For architectures in which the node order is significant, nodes will be considered consecutive in the order defined. For example, if the configuration for NodeName=charlie immediately follows the configuration for NodeName=baker they will be considered adjacent in the computer.

Feature
    A comma delimited list of arbitrary strings indicative of some characteristic associated with the node. There is no value associated with a feature at this time, a node either has a feature or it does not. If desired a feature may contain a numeric component indicating, for example, processor speed. By default a node has no features.

NodeAddr
    Name that a node should be referred to in establishing a communications path. This name will be used as an argument to the gethostbyname() function for identification. For example, "elx0012" might be used to designate the ethernet address for node "lx0012". A simple node range expression may optionally be used to specify ranges of nodes. The node range expression can contain one pair of square brackets with a sequence of comma separated numbers and/or ranges of numbers separated by a "-" (e.g. "elinux[0-64,128]"). If a node range expression is used to designate multiple nodes, they must exactly match the entries in the NodeName (e.g. "NodeName=lx[0-7] NodeAddr="elx[0-7]"). By default the NodeAddr will be identical in value to NodeName.

RealMemory
    Size of real memory on the node in MegaBytes (e.g. "2048"). The default value is 1.

Procs
    Number of processors on the node (e.g. "2"). The default value is 1.

Reason
    Identifies the reason for a node being in state "DOWN" or "DRAINED" or "DRAINING". Use quotes to enclose a reason having more than one word.

State
    State of the node with respect to the initiation of user jobs. Acceptable values are "BUSY", "DOWN", "DRAINED", "DRAINING", "IDLE", and "UNKNOWN". "BUSY" indicates the node has been allocated work and should not be used in the configuration file. "DOWN" indicates the node failed and is unavailable to be allocated work. "DRAINED" indicates the node was configured unavailable to be allocated work and is presently not performing any work. "DRAINING" indicates the node is unavailable to be allocated new work, but is completing the processing of a job. "IDLE" indicates the node available to be allocated work, but has none at present "UNKNOWN" indicates the node's state is undefined, but will be established when the slurmd daemon on that node registers. The default value is "UNKNOWN".

TmpDisk
    Total size of temporary disk storage in TmpFS in MegaBytes (e.g. "16384"). TmpFS (for "Temporary File System") identifies the location which jobs should use for temporary storage. Note this does not indicate the amount of free space available to the user on the node, only the total file system size. The system administration should insure this file system is purged as needed so that user jobs have access to most of this space. The Prolog and/or Epilog programs (specified in the configuration file) might be used to insure the file system is kept clean. The default value is 1.

Weight
    The priority of the node for scheduling purposes. All things being equal, jobs will be allocated the nodes with the lowest weight which satisfies their requirements. For example, a heterogeneous collection of nodes might be placed into a single partition for greater system utilization, responsiveness and capability. It would be preferable to allocate smaller memory nodes rather than larger memory nodes if either will satisfy a job's requirements. The units of weight are arbitrary, but larger weights should be assigned to nodes with more processors, memory, disk space, higher processor speed, etc. Weight is an integer value with a default value of 1.

The partition configuration permits you to establish different job limits or access controls for various groups (or partitions) of nodes. Nodes may be in only one partition. Jobs are allocated resources within a single partition. The partition configuration file contains the following information:

AllowGroups
    Comma separated list of group IDs which may use the partition. If at least one group associated with the user submitting the job is in AllowGroups, he will be permitted to use this partition. The default value is "ALL".

Default
    If this keyword is set, jobs submitted without a partition specification will utilize this partition. Possible values are "YES" and "NO". The default value is "NO".

Hidden
    Specifies if the partition and its jobs are to be hidden by default. Hidden partitions will by default not be reported by the SLURM APIs or commands. Possible values are "YES" and "NO". The default value is "NO".

RootOnly
    Specifies if only user ID zero (or user <i>root</i> may initiate jobs in this partition. Possible values are "YES" and "NO". The default value is "NO".

MaxNodes
    Maximum count of nodes which may be allocated to any single job. The default value is "UNLIMITED", which is represented internally as -1. This limit does not apply to jobs executed by SlurmUser or user root.

MaxTime
    Maximum wall-time limit for any job in minutes. The default value is "UNLIMITED", which is represented internally as -1. This limit does not apply to jobs executed by SlurmUser or user root.

MinNodes
    Minimum count of nodes which may be allocated to any single job. The default value is 1. This limit does not apply to jobs executed by SlurmUser or user root.

Nodes
    Comma separated list of nodes which are associated with this partition. Node names may be specified using the node range expression syntax described above. A blank list of nodes (i.e. "Nodes= ") can be used if one wants a partition to exist, but have no resources (possibly on a temporary basis).

PartitionName
    Name by which the partition may be referenced (e.g. "Interactive"). This name can be specified by users when submitting jobs.

Shared
    Ability of the partition to execute more than one job at a time on each node. Shared nodes will offer unpredictable performance for application programs, but can provide higher system utilization and responsiveness than otherwise possible. Possible values are "FORCE", "YES", and "NO". "FORCE" makes all nodes in the partition available for sharing without user means of disabling it. "YES" makes nodes in the partition available for sharing if and only if the individual jobs permit sharing (see the srun "--shared" option). "NO" makes nodes unavailable for sharing under all circumstances. The default value is "NO".

State
    State of partition or availability for use. Possible values are "UP" or "DOWN". The default value is "UP".

EXAMPLE

#
# Sample /etc/slurm.conf for dev[0-25].llnl.gov
# Author: John Doe
# Date: 11/06/2001
#
ControlMachine=dev0 ControlAddr=edev0
BackupController=dev1 BackupAddr=edev1
#
AuthType=auth/authd
Epilog=/usr/local/slurm/epilog
Prolog=/usr/local/slurm/prolog
FastSchedule=1
FirstJobId=65536
HashBase=10
HeartbeatInterval=60
InactiveLimit=120
JobCompType=jobcomp/filetxt
JobCompLoc=/var/log/slurm.job.log
KillWait=30
MaxJobCount=10000
MinJobAge=3600
PluginDir=/usr/local/lib:/usr/local/slurm/lib
ReturnToService=0
SchedulerType=sched/wiki
SchedulerAuth=42 SchedulerPort=7004
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdLogFile=/var/log/slurmd.log
SlurmctldDebug=4 SlurmdDebug=3
SlurmctldPort=7002 SlurmdPort=7003
SlurmctldTimeout=300 SlurmdTimeout=300
SlurmdSpoolDir=/usr/local/slurm/slurmd.spool
StateSaveLocation=/usr/local/slurm/slurm.state
SwitchType=switch/elan
TmpFS=/tmp
WaitTime=30
JobCredentialPrivateKey=/usr/local/slurm/private.key
JobCredentialPublicCertificate=/usr/local/slurm/public.cert
#
# Node Configurations
#
NodeName=DEFAULT Procs=2 RealMemory=2000 TmpDisk=64000
NodeName=DEFAULT State=UNKNOWN
NodeName=dev[0-25] NodeAddr=edev[0-25] Weight=16
# Update records for specific DOWN nodes
NodeName=dev20 State=DOWN Reason="power,ETA=Dec25"
#
# Partition Configurations
#
PartitionName=DEFAULT MaxTime=30 MaxNodes=10
PartitionName=debug Nodes=dev[0-8,18-25] State=UP Default=YES
PartitionName=batch Nodes=dev[9-17] State=UP MinNodes=4

COPYING
Copyright (C) 2002 The Regents of the University of California. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). UCRL-CODE-2002-040.

This file is part of SLURM, a resource management program. For details, see <http://www.llnl.gov/linux/slurm/>.

SLURM is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

SLURM is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

FILES
/etc/slurm.conf

SEE ALSO

gethostbyname(3), group(5), hostname(1), scontrol(1), slurmctld(8), slurmd(8), syslog(2)