# Burst Buffer Guide

DRAFT based upon early design work

Slurm version 15.08 includes support for burst buffers, a shared high-speed storage resource. Slurm provides support for allocating these resources, staging files in, scheduling compute nodes for jobs using these resources, then staging files out. This support is provided using a plugin mechanism so that a various burst buffer infrastructures may be easily used. Two plugins are provided initially:

1. cray - Uses Cray APIs to perform underlying management functions
2. generic - Uses scripts to perform underlying management functions

Additional plugins may be provided in future releases of Slurm.

Slurm's mode of operation follows this general pattern:

0. Read configuration information and initial state information
1. After expected start times for pending jobs are established, allocate burst buffers to those jobs expected to start earliest and start stage-in of required files
2. Jobs can be allocated compute nodes and begin execution after required burst buffer has been allocated and file stage-in has completed
3. After job has completed execution, begin file stage-out from burst buffer
4. After file stage-out has completed, burst buffer can be released and the job record purged.

## Configuration

Burst buffer support in Slurm is enabled by specifying the plugin(s) to loaded for managing these resources using the BurstBufferType configuration parameter in the slurm.conf file. Multiple plugin names may be specified in a comma separated list. Detailed logging of burst buffer specific actions may be generated by using the DebugFlags=BurstBuffer configuration parameter. This DebugFlag can result in very verbose logging and is only intended for diagnostic purposes rather than for use in a production system.

    # Excerpt of slurm.conf file
    BurstBufferType=burst_buffer/generic
    DebugFlags=BurstBuffer

Burst buffer specific options may be defined in a burst_buffer.conf file. This file can contain information about who can or can not use burst buffers, resource limits, priority boost information for jobs using burst buffers, timeouts, and paths to script used to perform various functions for the burst_buffer/generic plugin. Note that sample scripts for performing these actions are included with the Slurm distribution. These scripts were intended for Slurm development and testing purposes and are not intended for customer use. Contributions of usable scripts/programs are invited. For more information about the functionality that might be included in the scripts, please see The Ramdisk Storage Accelerator. Note that these scripts/programs are executed as SlurmUser rather than user root. Please see the burst_buffer.conf man page for more information.

    # Excerpt of burst_buffer.conf file
    AllowUsers=alan:brenda
    #
    JobSizeLimit=20GB   # Applies to each job
    UserSizeLimit=50GB  # Applies to ALL users
    #
    PrioBoostUse=100
    PrioBoostAlloc=200
    #
    StageInTimeout=30
    StageOutTimeout=30
    #
    GetSysState=/usr/local/slurm/15.08/sbin/GSS
    StartStageIn=/usr/local/slurm/15.08/sbin/SSI
    StartStageOut=/usr/local/slurm/15.08/sbin/SSO
    StopStageIn=/usr/local/slurm/15.08/sbin/PSI
    StopStageOut=/usr/local/slurm/15.08/sbin/PSO

## User Commands

Slurm's current burst buffer state information is available using the scontrol command. We intend to make this information available from the sview command at a later time.

    $ scontrol show burst
    Name=burst_buffer/generic TotalSpace=50GB UsedSpace=42GB
      JobSizeLimit=20GB UserSizeLimit=49GB
      PrioBoostAlloc=200 PrioBoostUse=100
      StageInTimeout=30 StageOutTimeout=30
      AllowUsers=alan:brenda
      GetSysState=/usr/local/slurm/15.08/sbin/GSS
      StartStageIn=/usr/local/slurm/15.08/sbin/SSI
      StartStageIn=/usr/local/slurm/15.08/sbin/SSO
      StopStageIn=/usr/local/slurm/15.08/sbin/PSI
      StopStageIn=/usr/local/slurm/15.08/sbin/PSO
        JobID=18 Size=10GB State=staged-in StateTime=2014-11-19T09:17:21 UserID=alan(1000)
        JobID=20 Size=10GB State=staged-in StateTime=2014-11-19T09:10:42 UserID=alan(1000)
        Name=DB1 Size=22GB State=staged-in StateTime=2014-11-19T09:02:21 UserID=brenda(1001)

All of the job submission commands (salloc, sbatch and srun) accept an option identifying the size of burst buffer required by the job (e.g. sbatch --bb=size=10gb my.bash). In addition, batch jobs can have files staged in and/or out by specifying those details in the batch script (details not yet available). (There will probably be a job_submit plugin for Cray systems that reads the user's batch script and sets the "-bb-" option for the job with the required spaced.)

Last modified 19 November 2014