# Large Cluster Administration Guide

This document contains SLURM administrator information specifically for clusters containing 1,024 nodes or more. Virtually all SLURM components have been validated (through emulation) for clusters containing up to 65,536 compute nodes. Getting optimal performance at that scale does require some tuning and this document should help you off to a good start. A working knowledge of SLURM should be considered a prerequisite for this material.

## Performance Results

SLURM has been used on clusters containing up to 4,184 nodes. At that scale, the total time to execute a simple program (resource allocation, task launch, I/O processing, and cleanup, e.g. "time srun -N4184 -n8368 uname") at 8,368 tasks across the 4,184 nodes was under 57 seconds. The table below shows total execution times for several large clusters with different architectures.

SLURM Total Job Execution Time
Nodes | Tasks | Seconds
--- | --- | ---
256 | 512 | 1.0
512 | 1024 | 2.2
1024 | 2048 | 3.7
2123 | 4246 | 19.5
4184 | 8368 | 56.6

## Node Selection Plugin (SelectType)

While allocating individual processors within a node is great for smaller clusters, the overhead of keeping track of the individual processors and memory within each node adds significant overhead. For best scalability, allocate whole nodes using select/linear or select/bluegene and avoid select/cons_res.

## Job Accounting Gather Plugin (JobAcctGatherType)

Job accounting relies upon the slurmstepd daemon on each compute node periodically sampling data. This data collection will take compute cycles away from the application inducing what is known as system noise. For large parallel applications, this system noise can detract for application scalability. For optimal application performance, disabling job accounting is best (jobacct_gather/none). Consider use of job completion records (JobCompType) for accounting purposes as this entails far less overhead. If job accounting is required, configure the sampling interval to a relatively large size (e.g. JobAcctGatherFrequency=300). Some experimentation may also be required to deal with collisions on data transmission.

## Node Configuration

While SLURM can track the amount of memory and disk space actually found on each compute node and use it for scheduling purposes, this entails extra overhead. Optimize performance by specifying the expected configuration using the available parameters (RealMemory, Procs, and TmpDisk). If the node is found to contain less resources than configured, it will be marked DOWN and not used. Also set the FastSchedule parameter. While SLURM can easily handle a heterogeneous cluster, configuring the nodes using the minimal number of lines in slurm.conf will both make for easier administration and better performance.

## Timers

If a high throughput of jobs is anticipated (i.e. large numbers of jobs with brief execution times) then configure MinJobAge to the smallest interval practical for your environment. MinJobAge specifies the minimum number of seconds that a terminated job will be retained by SLURM's control daemon before purging. After this time, information about terminated jobs will only be available through accounting records.

The configuration parameter SlurmdTimeout determines the interval at which slurmctld routinely communicates with slurmd. Communications occur at half the SlurmdTimeout value. The purpose of this is to determine when a compute node fails and thus should not be allocated work. Longer intervals decrease system noise on compute nodes (we do synchronize these requests across the cluster, but there will be some impact upon applications). For really large clusters, SlurmdTimeout values of 120 seconds or more are reasonable.

If MPICH-2 is used, the srun command will manage the key-pairs used to bootstrap the application. Depending upon the processor speed and architecture, the communication of key-pair information may require extra time. This can be done by setting an environment variable PMI_TIME before executing srun to launch the tasks. The default value of PMI_TIME is 500 and this is the number of microseconds alloted to transmit each key-pair. We have executed up to 16,000 tasks with a value of PMI_TIME=4000.

The individual slurmd daemons on compute nodes will initiate messages to the slurmctld daemon only when they start up or when the epilog completes for a job. When a job allocated a large number of nodes completes, it can cause a very large number of messages to be sent by the slurmd daemons on these nodes to the slurmctld daemon all at the same time. In order to spread this message traffic out over time and avoid message loss, The EpilogMsgTime parameter may be used. Note that even if messages are lost, they will be retransmitted, but this will result in a delay for reallocating resources to new jobs.

## Other

SLURM uses hierarchical communications between the slurmd daemons in order to increase parallelism and improve performance. The TreeWidth configuration parameter controls the fanout of messages. The default value is 50, meaning each slurmd daemon can communicate with up to 50 other slurmd daemons and over 2500 nodes can be contacted with two message hops. The default value will work well for most clusters. Optimal system performance can typically be achieved if TreeWidth is set to the square root of the number of nodes in the cluster for systems having no more than 2500 nodes or the cube root for larger systems.

The srun command automatically increases its open file limit to the hard limit in order to process all of the standard input and output connections to the launched tasks. It is recommended that you set the open file hard limit to 8192 across the cluster.

Last modified 13 May 2010