# Power Saving Guide

SLURM provides an integrated power saving mechanism for idle nodes. Nodes that remain idle for an configurable period of time can be placed in a power saving mode. The nodes will be restored to normal operation once work is assigned to them. Beginning with version 2.0.0, nodes can be fully powered down. Earlier releases of SLURM do not support the powering down of nodes, only support of reducing their performance and thus their power consumption. For example, power saving can be accomplished using a cpufreq governor that can change CPU frequency and voltage (note that the cpufreq driver must be enabled in the Linux kernel configuration). Of particular note, SLURM can power nodes up or down at a configurable rate to prevent rapid changes in power demands. For example, starting a 1000 node job on an idle cluster could result in an instantaneous surge in power demand of multiple megawatts without SLURM's support to increase power demands in a gradual fashion.

## Configuration

A great deal of flexibility is offered in terms of when and how idle nodes are put into or removed from power save mode. The following configuration parameters are available:

- SuspendTime: Nodes becomes eligible for power saving mode after being idle for this number of seconds. The configured value should exceed the time to suspend and resume a node. A negative number disables power saving mode. The default value is -1 (disabled).
- SuspendRate: Maximum number of nodes to be placed into power saving mode per minute. A value of zero results in no limits being imposed. The default value is 60. Use this to prevent rapid drops in power requirements.
- ResumeDelay: Minimum delay between when a node is suspended before attempting to resume it (e.g. power it back up). This value should exceed the time required for SuspendProgram to fully suspend a node. Its value should be no larger than SlurmdTimeout. The default value is 30 seconds.
- ResumeRate: Maximum number of nodes to be removed from power saving mode per minute. A value of zero results in no limits being imposed. The default value is 300. Use this to prevent rapid increases in power requirements.
- SuspendProgram: Program to be executed to place nodes into power saving mode. The program executes as SlurmUser (as configured in slurm.conf). The argument to the program will be the names of nodes to be placed into power savings mode (using SLURM's hostlist expression format).
- ResumeProgram: Program to be executed to remove nodes from power saving mode. The program executes as SlurmUser (as configured in slurm.conf). The argument to the program will be the names of nodes to be removed from power savings mode (using SLURM's hostlist expression format). This program may use the scontrol show node command to insure that a node has booted and the slurmd daemon started. If the slurmd daemon fails to respond within the configured SlurmdTimeout value, the node will be placed in a DOWN state and the job requesting the node will be requeued. For reasons of reliability, ResumeProgram may execute more than once for a node when the slurmctld daemon is restarted or reconfigured.
- SuspendExcNodes: List of nodes to never place in power saving mode. Use SLURM's hostlist expression format. By default, no nodes are excluded.
- SuspendExcParts: List of partitions with nodes to never place in power saving mode. Multiple partitions may be specified using a comma separator. By default, no nodes are excluded.

Note that SuspendProgram and ResumeProgram execute as SlurmUser on the node where the slurmctld daemon runs (primary and backup server nodes). Use of sudo may be required for SlurmUserto power down and restart nodes. If you need to convert SLURM's hostlist expression into individual node names, the scontrol show hostnames command may prove useful. The commands used to boot or shut down nodes will depend upon your cluster management tools.

Note that SuspendProgram and ResumeProgram are not subject to any time limits. They should perform the required action, ideally verify the action (e.g. node boot and slurmd daemon started and thus the node is no longer non-responsive to slurmctld), and terminate. Long running programs will be logged by slurmctld, but not aborted.

    #!/bin/bash
    # Example SuspendProgram
    hosts=`scontrol show hostnames $1`
    for host in "$hosts"
    do
       sudo node_shutdown $host
    done

    #!/bin/bash
    # Example ResumeProgram
    hosts=`scontrol show hostnames $1`
    for host in "$hosts"
    do
       sudo node_startup $host
    done

Subject to the various rates, limits and exclusions, the power save code follows this logic:

1. Identifiy nodes which have been idle for at least SuspendTime.
2. Execute SuspendProgram with an argument of the idle node names.
3. Identify the nodes which are in power save mode (a flag in the node's state field), but have been allocated to jobs.
4. Execute ResumeProgram with an argument of the allocated node names.
5. Once the slurmd responds, initiate the job and/or job steps allocated to it.
6. If the slurmd fails to respond within the value configured for SlurmdTimeout, the node will be marked DOWN and the job requeued if possible.
7. Repeat indefinitely.

The slurmctld daemon will periodically (every 10 minutes) log how many nodes are in power save mode using messages of this sort:

    [May 02 15:31:25] Power save mode 0 nodes
    ...
    [May 02 15:41:26] Power save mode 10 nodes
    ...
    [May 02 15:51:28] Power save mode 22 nodes

Using these logs you can easily see the effect of SLURM's power saving support. You can also configure SLURM without SuspendProgram or ResumeProgram values to assess the potential impact of power saving mode before enabling it.

Last modified 13 May 2009