# Generic Resource (GRES) Scheduling

## Contents

Overview
Configuration
Running Jobs
GPU Management
MPS Management
MIC Management

## Overview

Generic resource (GRES) scheduling is supported through a flexible plugin mechanism. Support is currently provided for Graphics Processing Units (GPUs), CUDA Multi-Process Service (MPS), and Intel® Many Integrated Core (MIC) processors.

## Configuration

Slurm supports no generic resources in the default configuration. One must explicitly specify which resources are to be managed in the slurm.conf configuration file. The configuration parameters of interest are:

- GresTypes a comma delimited list of generic resources to be managed (e.g. GresTypes=gpu,mps). This name may be that of an optional plugin providing additional control over the resources.
- Gres the generic resource configuration details in the format
  <name>[:<type>][:no_consume]:<number>[K|M|G]
  The first field is the resource name, which matches the GresType configuration parameter name. The optional type field might be used to identify a model of that generic resource. A generic resource can also be specified as non-consumable (i.e. multiple jobs can use the same generic resource) with the optional field ":no_consume". The final field must specify a generic resource count. A suffix of "K", "M" or "G" may be used to multiply the count by 1024, 1048576 or 1073741824 respectively. By default a node has no generic resources.

Note that the GRES specification for each node works in the same fashion as the other resources managed. Depending upon the value of the FastSchedule parameter, nodes which are found to have fewer resources than configured will be placed in a DOWN state.

Sample slurm.conf file:

    # Configure support for our four GPUs
    GresTypes=gpu,mps,bandwidth
    NodeName=tux[0-7] Gres=gpu:tesla:2,gpu:kepler:2,mps:400,bandwidth:lustre:no_consume:4G

Each compute node with generic resources typically contain a gres.conf file describing which resources are available on the node, their count, associated device files and cores which should be used with those resources. In the case of GPUs, their configuration will be automatically gathered using the NVML library, if installed. Configuration information about all other generic resource must explicitly be described in the gres.conf file. The configuration parameters available are:

- Name name of a generic resource (must match GresTypes values in slurm.conf).
- Count Number of resources of this type available on this node. The default value is set to the number of File values specified (if any), otherwise the default value is one. A suffix of "K", "M" or "G" may be used to multiply the number by 1024, 1048576 or 1073741824 respectively (e.g. "Count=10G"). Note that Count is a 32-bit field and the maximum value is 4,294,967,295.
- Cores Specify the first thread CPU index numbers for the specific cores which can use this resource. For example, it may be strongly preferable to use specific cores with specific devices (e.g. on a NUMA architecture). Multiple cores may be specified using a comma delimited list or a range may be specified using a "-" separator (e.g. "0,1,2,3" or "0-3"). If specified, then only the identified cores can be allocated with each generic resource; an attempt to use other cores will not be honored. If not specified, then any core can be used with the resources, which also increases the speed of Slurm's scheduling algorithm. If any core can be effectively used with the resources, then do not specify the Cores option for improved speed in the Slurm scheduling logic. NOTE: If your cores contain multiple threads only list the first thread of each core. The logic is such that it uses core instead of thread scheduling per GRES. Also note that since Slurm must be able to perform resource management on heterogeneous clusters having various core ID numbering schemes, an abstract index will be used instead of the physical core index. That abstract id may not correspond to your physical core number. Basically Slurm starts numbering from 0 to n, being 0 the id of the first processing unit (core or thread if HT is enabled) on the first socket, first core and maybe first thread, and then continuing sequentially to the next thread, core, and socket. The numbering generally coincides with the processing unit logical number (PU L#) seen in lstopo output.
- File Fully qualified pathname of the device files associated with a resource. The name can include a numeric range suffix to be interpreted by Slurm (e.g. File=/dev/nvidia[0-3]). This field is generally required if enforcement of generic resource allocations are to be supported (i.e. prevents a user from making use of resources allocated to a different user). Enforcement of the file allocation relies upon Linux Control Groups (cgroups) and Slurm's task/cgroup plugin, which will place the allocated files into the job's cgroup and prevent use of other files. Please see Slurm's Cgroups Guide for more information.
  Except in the case of MPS support, if File is specified then Count must be either set to the number of file names specified or not set (the default value is the number of files specified). In the case of MPS support, each GPU would be identifed by name using the File parameter and Count would specify the number of MPS entries that would correspond to that GPU; typically 100 or some multiple of 100.
  NOTE: If you specify the File parameter for a resource on some node, the option must be specified on all nodes and Slurm will track the assignment of each specific resource on each node. Otherwise Slurm will only track a count of allocated resources rather than the state of each individual device file.
  NOTE: Drain a node before changing the count of records with File parameters (i.e. if you want to add or remove GPUs from a node's configuration). Failure to do so will result in any job using those GRES being aborted.
- Type Optionally specify the device type. For example, this might be used to identify a specific model of GPU, which users can then specify in their job request. If Type is specified, then Count is limited in size (currently 1024).

Sample gres.conf file:

    # Configure support for four GPUs (with MPS), plus bandwidth
    Name=gpu Type=gp100  File=/dev/nvidia0 Cores=0,1
    Name=gpu Type=gp100  File=/dev/nvidia1 Cores=0,1
    Name=gpu Type=p6000  File=/dev/nvidia2 Cores=2,3
    Name=gpu Type=p6000  File=/dev/nvidia3 Cores=2,3
    Name=mps Count=200  File=/dev/nvidia0
    Name=mps Count=200  File=/dev/nvidia1
    Name=mps Count=100  File=/dev/nvidia2
    Name=mps Count=100  File=/dev/nvidia3
    Name=bandwidth Type=lustre Count=4G

## Running Jobs

Jobs will not be allocated any generic resources unless specifically requested at job submit time using the options:

--gres
  Generic resources required per node
--gpu
  GPUs required per job
--gpu-per-node
  GPUs required per node. Equivalent to the --gres option for GPUs.
--gpu-per-socket
  GPUs required per socket. Requires the job to specify a task socket.
--gpu-per-task
  GPUs required per task. Requires the job to specify a task count.

All of these options are supported by the salloc, sbatch and srun commands. Note that all of the --gpu* options are only supported by Slurm's select/cons_tres plugin. Jobs requesting these options when the select/cons_tres plugin is not configured will be rejected. The --gres option requires an argument specifying which generic resources are required and how many resources using the form name[:type:count] while all of the --gpu* options require an argument of the form [type]:count. The name is the same name as specified by the GresTypes and Gres configuration parameters. type identifies a specific type of that generic resource (e.g. a specific model of GPU). count specifies how many resources are required and has a default value of 1. For example:
sbatch --gres=gpu:kepler:2 ....

Several addition resource requirement specifications are available specifically for GPUs and detailed descriptions about these options are available in the man pages for the job submission commands. As for the --gpu* option, these options are only supported by Slurm's select/cons_tres plugin.

--cpus-per-gpu
  Count of CPUs allocated per GPU.
--gpu-bind
  Define how tasks are bound to GPUs.
--gpu-freq
  Specify GPU frequency and/or GPU memory frequency.
--mem-per-gpu
  Memory allocated per GPU.

Jobs will be allocated specific generic resources as needed to satisfy the request. If the job is suspended, those resources do not become available for use by other jobs.

Job steps can be allocated generic resources from those allocated to the job using the --gres option with the srun command as described above. By default, a job step will be allocated all of the generic resources allocated to the job. If desired, the job step may explicitly specify a different generic resource count than the job. This design choice was based upon a scenario where each job executes many job steps. If job steps were granted access to all generic resources by default, some job steps would need to explicitly specify zero generic resource counts, which we considered more confusing. The job step can be allocated specific generic resources and those resources will not be available to other job steps. A simple example is shown below.

    #!/bin/bash
    #
    # gres_test.bash
    # Submit as follows:
    # sbatch --gres=gpu:4 -n4 -N1-1 gres_test.bash
    #
    srun --gres=gpu:2 -n2 --exclusive show_device.sh &
    srun --gres=gpu:1 -n1 --exclusive show_device.sh &
    srun --gres=gpu:1 -n1 --exclusive show_device.sh &
    wait

## GPU Management

In the case of Slurm's GRES plugin for GPUs, the environment variable `CUDA_VISIBLE_DEVICES` is set for each job step to determine which GPUs are available for its use on each node. This environment variable is only set when tasks are launched on a specific compute node (no global environment variable is set for the salloc command and the environment variable set for the sbatch command only reflects the GPUs allocated to that job on that node, node zero of the allocation). CUDA version 3.1 (or higher) uses this environment variable in order to run multiple jobs or job steps on a node with GPUs and ensure that the resources assigned to each are unique. In the example above, the allocated node may have four or more graphics devices. In that case, `CUDA_VISIBLE_DEVICES` will reference unique devices for each file and the output might resemble this:

    JobStep=1234.0 CUDA_VISIBLE_DEVICES=0,1
    JobStep=1234.1 CUDA_VISIBLE_DEVICES=2
    JobStep=1234.2 CUDA_VISIBLE_DEVICES=3

NOTE: Be sure to specify the File parameters in the gres.conf file and ensure they are in the increasing numeric order.

When possible, Slurm automatically determines the GPUs on the system using NVIDIA's NVML library. The NVML library (which powers the `nvidia-smi` tool) numbers GPUs in order by their PCI bus IDs. For this numbering to match the numbering reported by CUDA, the `CUDA_DEVICE_ORDER` environmental variable must be set to `CUDA_DEVICE_ORDER=PCI_BUS_ID`.

GPU device files (e.g. /dev/nvidia1) are based on the Linux minor number assignment, while NVML's device numbers are assigned via PCI bus ID, from lowest to highest. Mapping between these two is indeterministic and system dependent, and could vary between boots after hardware or OS changes. For the most part, this assignment seems fairly stable. However, an after-bootup check is required to guarantee that a GPU device is assigned to a specific device file.

Please consult the NVIDIA CUDA documentation for more information about the `CUDA_VISIBLE_DEVICES` and `CUDA_DEVICE_ORDER` environmental variables.

## MPS Management

CUDA Multi-Process Service (MPS) provides a mechanism where GPUs can be shared by multiple jobs, where each job is allocated some percentage of the GPU's resources. GPUs to be made available for MPS should be identified in the slurm.conf and gres.conf configuration files with each device File being explicitly named and its Count specified (typically a multiple of 100, although different values can be used for different GPU types in a heterogeneous environment to reflect the processing power of each). Alternately, specify Count without File and the value of Count will be distributed evenly across all defined GPUs. Note that if NVIDIA's NVML library is installed, the GPU configuration (i.e. Type, File, Cores and Links data) will be automatically gathered from there and need not be recorded in the gres.conf file. If the GPU sharing that File name contains Core or Type parameters, that information will be automatically copied to the corresponding MPS GRES. Note that the same GPU can be allocated either as a GPU type GRES or as an MPS type GRES, but not both. In other words, once a GPU has been allocated as a gres/gpu resource it will not be available as a gres/mps. Likewise, once a GPU has been allocated as a gres/mps resource it will not be available as a gres/gpu. However the same GPU can be allocated as MPS generic resources to multiple jobs so long as the total count of MPS allocated to jobs does not exceed the configured count. Two example configurations for Slurm's gres.conf file are shown below.

    # Example 1 of gres.conf
    # Configure support for four GPUs (with MPS), plus bandwidth
    Name=gpu Type=gp100 File=/dev/nvidia0 Cores=0,1
    Name=gpu Type=gp100 File=/dev/nvidia1 Cores=0,1
    Name=gpu Type=p6000 File=/dev/nvidia2 Cores=2,3
    Name=gpu Type=p6000 File=/dev/nvidia3 Cores=2,3
    # Set gres/mps Count value to 100 on each of the 4 available GPUs
    Name=mps Count=400
    Name=bandwidth Type=lustre Count=4G

    # Example 2 of gres.conf
    # Configure support for four differernt GPU types (with MPS)
    Name=gpu Type=gtx1080 File=/dev/nvidia0 Cores=0,1
    Name=gpu Type=gtx1070 File=/dev/nvidia1 Cores=0,1
    Name=gpu Type=gtx1060 File=/dev/nvidia2 Cores=2,3
    Name=gpu Type=gtx1050 File=/dev/nvidia3 Cores=2,3
    Name=mps Count=1300   File=/dev/nvidia0
    Name=mps Count=1200   File=/dev/nvidia1
    Name=mps Count=1100   File=/dev/nvidia2
    Name=mps Count=1000   File=/dev/nvidia3

NOTE: Slurm support for gres/mps requires the use of the select/cons_tres plugin.

Job requests for MPS will be processed the same as any other GRES except that the request must be satisfied using only one GPU per node. For example, a job request for "--gres=mps:50" will not be satisfied by using 20 percent of one GPU and 30 percent of a second GPU on a single node. Multiple jobs from different users can use MPS on a node at the same time, but only one job for a given user can use MPS on a node at any given time. However the user can utilize MPS resources on other nodes as appropriate. Also note that GRES types of GPU and MPS can not be requested within a single job. Also jobs requesting MPS resources can not specify a GPU frequency. An epilog program should be used to terminate any MPS server started by the user/job.

Jobs requesting MPS resources will have the same environment variables set as for GPU resources, namely `CUDA_VISIBLE_DEVICES` and `CUDA_DEVICE_ORDER`. In addition they will have the `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE` environment variable set where its value is the job's percentage of MPS resources available on the assigned GPU. The percentage will be calculated based upon the portion of the configured Count on the Gres is allocated to a job of step. For example, a job requesting "--gres=gpu:200" and using the second configuration above would be allocated
15% of the gtx1080 (File=/dev/nvidia0, 200 x 100 / 1300 = 15), or
16% of the gtx1070 (File=/dev/nvidia0, 200 x 100 / 1200 = 16), or
18% of the gtx1060 (File=/dev/nvidia0, 200 x 100 / 1100 = 18), or
20% of the gtx1050 (File=/dev/nvidia0, 200 x 100 / 1000 = 20).

Please consult the NVIDIA Multi-Process Service documentation for more information about MPS.

## MIC Management

Slurm can be used to provide resource management for systems with the Intel® Many Integrated Core (MIC) processor. Slurm sets an `OFFLOAD_DEVICES` environment variable, which controls the selection of MICs available to a job step. The `OFFLOAD_DEVICES` environment variable is used by both Intel LEO (Language Extensions for Offload) and the MKL (Math Kernel Library) automatic offload. (This is very similar to how the `CUDA_VISIBLE_DEVICES` environment variable is used to control which GPUs can be used by CUDA™ software.) If no MICs are reserved via GRES, the `OFFLOAD_DEVICES` variable is set to -1. This causes the code to ignore the offload directives and run MKL routines on the CPU. The code will still run but only on the CPU. This also gives a somewhat cryptic warning:

    offload warning: OFFLOAD_DEVICES device number -1 does not correspond
    to a physical device

The offloading is automatically scaled to all the devices, (e.g. if --gres=mic:2 is defined) then all offloads use two MICs unless explicitly defined in the offload pragmas.

Last modified 15 January 2019