# Resource Limits

SLURM scheduling policy support was significantly changed in version 2.0 in order to take advantage of the database integration used for storing accounting information. This document describes the capabilities available in SLURM version 2.0. New features are under active development. Familiarity with SLURM's Accounting web page is strongly recommended before use of this document.

Note for users of Maui or Moab schedulers:
Maui and Moab are not integrated with SLURM's resource limits, but should use their own resource limits mechanisms.

## Configuration

Scheduling policy information must be stored in a database as specified by the AccountingStorageType configuration parameter in the slurm.conf configuration file. Information can be recorded in either MySQL or PostgreSQL. For security and performance reasons, the use of SlurmDBD (SLURM Database Daemon) as a front-end to the database is strongly recommended. SlurmDBD uses a SLURM authentication plugin (e.g. Munge). SlurmDBD also uses an existing SLURM accounting storage plugin to maximize code reuse. SlurmDBD uses data caching and prioritization of pending requests in order to optimize performance. While SlurmDBD relies upon existing SLURM plugins for authentication and database use, the other SLURM commands and daemons are not required on the host where SlurmDBD is installed. Only the slurmdbd and slurm-plugins RPMs are required for SlurmDBD execution.

Both accounting and scheduling policy are configured based upon an association. An association is a 4-tuple consisting of the cluster name, bank account, user and (optionally) the SLURM partition. In order to enforce scheduling policy, set the value of AccountingStorageEnforce to "1" in slurm.conf. This prevents users from running any jobs without an valid association record in the database and enforces scheduling policy limits that have been configured. In order to enforce association limits along with scheduling policy, set the value of AccountingStorageEnforce to "2" in slurm.conf. When set to "1" association limits will not be enforced. It is a good idea to run in this mode when running a scheduler on top of slurm, like Moab, that does not update in real time their limits per association.

## Tools

The tool used to manage accounting policy is sacctmgr. It can be used to create and delete cluster, user, bank account, and partition records plus their combined association record. See man sacctmgr for details on this tools and examples of its use.

A web interface with graphical output is currently under development.

Changes made to the scheduling policy are uploaded to the SLURM control daemons on the various clusters and take effect immediately. When an association is delete, all jobs running or pending which belong to that association are immediately cancelled. When limits are lowered, running jobs will not be cancelled to satisfy the new limits, but the new lower limits will be enforced.

## Policies supported

A limited subset of scheduling policy options are currently supported. The available options are expected to increase as development continues. Most of these scheduling policy options are available not only for an association, but also for each cluster and account. If a new association is created for some user and some scheduling policy options is not specified, the default will be the option for the cluster plus account pair and if that is not specified then the cluster and if that is not specified then no limit will apply.

Currently available cheduling policy options:

- MaxJobs Maxiumum number of running jobs for this association
- MaxNodes Maxiumum number of nodes for any single jobs in this association
- MaxWall Maxiumum wall clock time limit for any single jobs in this association

The MaxNodes and MaxWall options already exist in SLURM's configuration on a per-partition basis, but these options provide the ability to establish limits on a per-user basis. The MaxJobs option provides an entirely new mechanism for SLURM to control the workload any individual may place on a cluster in order to achieve some balance between users.

The next scheduling policy expected to be added is the concept of fair-share scheduling based upon the hierarchical bank account data is already maintained in the SLURM database. The priorities of pending jobs will be adjusted in order to deliver resources in proportion to each association's fair-share. Consider the trivial example of a single bank account with two users named Alice and Brian. We might allocate Alice 60 percent of the resources and Brian the remaining 40 precent. If Alice has actually used 80 percent of available resources in the recent past, then Brian's pending jobs will automatically be given a higher priority than Alice's in order to deliver resources in proportion to the fair-share target. The time window considered in fair-share scheduling will be configurable as well as the relative importance of job age (time waiting to run), but this this example illustrates the concepts involved.

Last modified 9 February 2009