# Scheduling Configuration Guide

## Overview

Slurm is designed to perform a quick and simple scheduling attempt at frequent intervals (e.g. at the time of each job submission or completion). A more comprehensive scheduling attempt is made less frequently, typically by the backfill scheduling plugin.

## Scheduling Configuration

The SchedulerType configuration parameter specifies the scheduler plugin to use. Options are sched/backfill, which performs backfill scheduling, and sched/builtin, which attempts to schedule jobs in a string priority order within each partition/queue.

There is also a SchedulerParameters configuration parameter which can specify a wide range of parameters as described below. This first set of parameters applies to all scheduling configurations.

- default_queue_depth=# - Specifies the number of jobs to consider for scheduling on each event that may result in a job being scheduled. Default value is 100 jobs. Since this happens frequently, a relatively small number is generally best.
- defer - Do not attempt to schedule jobs individually at submit time. Can be useful for high-throughput computing.
- max_switch_wait=# - Specifies the maximum time a job can wait for desired number of leaf switches. Default value is 300 seconds.

## Backfill Scheduling

Backfill scheduling plugin is loaded by default. Without backfill scheduling, each partition is scheduled strictly in priority order, which typically results in significantly lower system utilization and responsiveness than otherwise possible. Backfill scheduling will start lower priority jobs if doing so does not delay the expected start time of any higher priority jobs. Since the expected start time of pending jobs depends upon the expected completion time of running jobs, reasonably accurate time limits are important for backfill scheduling to work well.

Slurm's backfill scheduler takes into consideration every running job. It then considers pending jobs in priority order, determining when and where each will start, taking into consideration the possibility of job preemption, gang scheduling, generic resource (GRES) requirements, memory requirements, etc. If the job under consideration can start immediately without impacting the expected start time of any higher priority job, then it does so. Otherwise the resources required by the job will be reserved during the job's expected execution time.

Backfill scheduling is difficult without reasonable time limit estimates for jobs, but some configuration parameters that can help.

- DefaultTime - Default job time limit (specify value by partition)
- MaxTime - Maximum job time limit (specify value by partition)
- OverTimeLimit - Amount by which a job can exceed its time limit before it is killed. A system-wide configuration parameter.

Backfill scheduling is a time consuming operation. Locks are released briefly every two seconds so that other options can be processed, for example to process new job submission requests. Backfill scheduling can optionally continue execution after the lock release and ignore newly submitted jobs (SchedulerParameters=bf_continue). Doing so will permit consideration of more jobs, but may result in the delayed scheduling of newly submitted jobs. A list of SchedulerParameters configuration parameters related to backfill scheduling follows.

- bf_continue - If set, then continue backfill scheduling after periodically releasing locks for other operations.
- bf_interval=# - Interval between backfill scheduling attempts. Default value is 30 seconds.
- bf_max_job_part=# - Maximum number of jobs to initiate per partition in each backfill cycle. Default value is 0 (no limit).
- bf_max_job_start=# - Maximum number of jobs to initiate in each backfill cycle. Default value is 0 (no limit).
- bf_max_job_test=# - Maximum number of jobs consider for backfill scheduling in each backfill cycle. Default value is 100 jobs.
- bf_max_job_user=# - Maximum number of jobs to initiate per user in each backfill cycle. Default value is 0 (no limit).
- bf_resolution=# - Time resolution of backfill scheduling. Default value is 60 seconds. Larger values are appropriate if job time limits are imprecise and/or small delays in starting pending jobs in order to achieve higher system utilization is desired.
- bf_window=# - How long into the future to look when determining when and where jobs can start. Default value is one day.

Last modified 27 March 2014