# Slurm Burst Buffer Guide

DRAFT based upon early design work

Slurm version 15.08 includes support for burst buffers, a shared high-speed storage resource. Slurm provides support for allocating these resources, staging files in, scheduling compute nodes for jobs using these resources, then staging files out. Burst buffers can also be used as temporary storage during a job's lifetime, without file staging. Another typical use case is for persist storage, not associated with any specific job. This support is provided using a plugin mechanism so that a various burst buffer infrastructures may be easily configured. Two plugins are provided initially:

1. cray - Uses Cray APIs to perform underlying management functions
2. generic - Uses system administrator defined scripts to perform underlying management functions

Additional plugins may be provided in future releases of Slurm.

Slurm's mode of operation follows this general pattern:

0. Read configuration information and initial state information
1. After expected start times for pending jobs are established, allocate burst buffers to those jobs expected to start earliest and start stage-in of required files
2. After stage-in has completed, jobs can be allocated compute nodes and begin execution
3. After job has completed execution, begin file stage-out from burst buffer
4. After file stage-out has completed, burst buffer can be released and the job record purged

## Configuration

Burst buffer support in Slurm is enabled by specifying the plugin(s) to loaded for managing these resources using the BurstBufferType configuration parameter in the slurm.conf file. Multiple plugin names may be specified in a comma separated list. Detailed logging of burst buffer specific actions may be generated by using the DebugFlags=BurstBuffer configuration parameter. This DebugFlag can result in very verbose logging and is only intended for diagnostic purposes rather than for use in a production system.

    # Excerpt of slurm.conf file
    BurstBufferType=burst_buffer/generic
    DebugFlags=BurstBuffer

Burst buffer specific options should be defined in a burst_buffer.conf file. This file can contain information about who can or can not use burst buffers, resource limits, priority boost information for jobs using burst buffers, timeouts, and paths to script used to perform various functions for the burst_buffer/generic plugin. Note that sample scripts for performing these actions are included with the Slurm distribution. These scripts were intended for Slurm development and testing purposes and are not intended for customer use. Contributions of usable scripts/programs are invited. For more information about the functionality that might be included in the scripts, please see The Ramdisk Storage Accelerator. Note that these scripts/programs are executed as SlurmUser rather than user root. Please see the burst_buffer.conf man page for more information.

    # Excerpt of burst_buffer.conf file for generic plugin
    AllowUsers=alan:brenda
    #
    JobSizeLimit=20GB   # Applies to each job
    UserSizeLimit=50GB  # Applies to ALL users
    #
    PrioBoostUse=100
    PrioBoostAlloc=200
    #
    StageInTimeout=30
    StageOutTimeout=30
    #
    GetSysState=/usr/local/slurm/15.08/sbin/GSS
    StartStageIn=/usr/local/slurm/15.08/sbin/SSI
    StartStageOut=/usr/local/slurm/15.08/sbin/SSO
    StopStageIn=/usr/local/slurm/15.08/sbin/PSI
    StopStageOut=/usr/local/slurm/15.08/sbin/PSO

Note for Cray systems: The JSON-C library must be installed in order to build Slurm's burst_buffer/cray plugin, which must parse JSON format data. See Slurm's JSON installation information for details.

## Job Submission Commands

All of the job submission commands (salloc, sbatch and srun) accept a "--bb" option to identify burst buffer requirements by the job. The normal mode of operation is for batch jobs to specify burst buffer requirements within the batch script rather than using the "--bb" option. Batch script lines containing a prefix of "#BB" identify the job's burst buffer space requirements, files to be staged in, files to be staged out, etc. when using the burst_buffer/generic plugin. The prefix of "#DW" (for "DataWarp") is used for burst buffer directives when using the burst_buffer/cray plugin. Interactive jobs (those submitted using the salloc and srun commands) can specify their burst buffer space requirements using the "--bb" option, but staging of jobs before and after the resource allocation is not supported (e.g. srun --bb=size=10tb my.bash).

Users may also request to be notified by email upon completion of burst buffer stage out using the "--mail-type=stage_out" or "--mail-type=all" option. The subject line of the email will be of this form:

    SLURM Job_id=12 Name=my_app Staged Out, StageOut time 00:05:07

A basic validation is performed on the DataWarp options at job submit time. If the DataWarp options are invalid, the job will be rejected and the error message generated by the Cray tools will be returned directly to the user. If the job is accepted, but later fails, the job will be held and its "Reason" field will be set to the error message generated by the Cray tools.

### User Options for Generic Plugin

TBD

### User Options for Cray Plugin

Note that the normal Slurm job submission commands are used (i.e. salloc, sbatch and srun). The burst buffer options in the batch script, identified with a prefix of "#DW", are passed to Cray software for processing. See the Cray documentation for details about those options.

Slurm also supports the creation and destruction of persistent burst buffers by using a prefix of "#BB" in the batch script. See Cray documentation for buffer access and type options. Samples of persistent burst buffer commands are shown below.

    #BB create_persistent  name=... capacity=# access=... type=...
    #BB destroy_persistent name=...
    #BB destroy_persistent name=... hurry

## Status Commands

Slurm's current burst buffer state information is available using the scontrol show burst command or by using the sview command's Burst Buffer tab.

    $ scontrol show burst
    Name=generic Granularity=100GB TotalSpace=50TB UsedSpace=42TB
      JobSizeLimit=10TB UserSizeLimit=40TB
      PrioBoostAlloc=200 PrioBoostUse=100
      StageInTimeout=30 StageOutTimeout=30
      AllowUsers=alan:brenda
      GetSysState=/usr/local/slurm/15.08/sbin/GSS
      StartStageIn=/usr/local/slurm/15.08/sbin/SSI
      StartStageIn=/usr/local/slurm/15.08/sbin/SSO
      StopStageIn=/usr/local/slurm/15.08/sbin/PSI
      StopStageIn=/usr/local/slurm/15.08/sbin/PSO
        JobID=18 Size=10TB State=staged-in StateTime=2014-11-19T09:17:21 UserID=alan(1000)
        JobID=20 Size=10TB State=staged-in StateTime=2014-11-19T09:10:42 UserID=alan(1000)
        Name=DB1 Size=22TB State=staged-in StateTime=2014-11-19T09:02:21 UserID=brenda(1001)

## Advanced Reservations

Burst buffer resources can be placed in an advanced reservation using the BurstBuffer option. The argument consists of four elements:
[plugin:][type:]#[units]
plugin is the burst buffer plugin name, currently either "cray" or "generic". If no plugin is specified, the reservation applies to all configured burst buffer plugins.
type specifies a Cray generic burst buffer resource, for example "nodes".
if "type" is not specified, the number is a measure of storage space.
units may be "N" (nodes), "GB" (gigabytes), "TB" (terabytes), "PB" (petabytes), etc. with the default units being megabyes for reservations of storage space.
Jobs using this reservation are not restricted to these burst buffer resources, but may use these reserved resources plus any which are generally available. Some examples follow.

    $ scontrol create reservation starttime=now duration=60 \
      users=alan flags=any_nodes \
      burstbuffer=cray:100GB,generic:20GB

    $ scontrol create reservation StartTime=noon duration=60 \
      users=brenda NodeCnt=8 \
      BurstBuffer=cray:nodes:2,generic:20GB

Last modified 28 July 2015