gres.conf(5)

NAME
gres.conf - Slurm configuration file for generic resource management.

DESCRIPTION
gres.conf is an ASCII file which describes the configuration of generic resources on each compute node. Each node must contain a gres.conf file if generic resources are to be scheduled by Slurm. The file location can be modified at system build time using the DEFAULT_SLURM_CONF parameter or at execution time by setting the SLURM_CONF environment variable. The file will always be located in the same directory as the slurm.conf file. If generic resource counts are set by the gres plugin function node_config_load(), this file may be optional.

Parameter names are case insensitive. Any text following a "#" in the configuration file is treated as a comment through the end of that line. Changes to the configuration file take effect upon restart of Slurm daemons, daemon receipt of the SIGHUP signal, or execution of the command "scontrol reconfigure" unless otherwise noted.

For "GresTypes=gpu", by default Slurm will use all GPUs detected on the system at slurmd daemon startup. To ignore specific GPUs, simply specify Ignore=true along with Name (=gpu) and File. Alternatively, you can choose to specify only the explicit GPUs you want to use. If you do this, other lines with Ignore=true are of no effect.

CUDA Multi-Process Service (MPS) provides a mechanism where GPUs can be shared by multiple jobs, where each job is allocated some percentage of the GPU's resources. GPUs to be made available for MPS must be identified in the slurm.conf and gres.conf configuration files with each device File being explicitly named and its Count specified (typically a multiple of 100). If the GPU sharing that File name contains Core or Type parameters, that information will be automatically copied to the corresponding MPS GRES (details about those configuration parameters below). Note that the same GPU can be allocated either as a GPU type GRES or as an MPS type GRES, but not both. In other words, once a GPU has been allocated as a gres/gpu resource it will not be available as a gres/mps. Likewise, once a GPU has been allocated as a gres/mps resource it will not be available as a gres/gpu.

NOTE: Slurm support for gres/mps requires the use of the select/cons_tres plugin.

The overall configuration parameters available include:

Count
    Number of resources of this type available on this node. The default value is set to the number of File values specified (if any), otherwise the default value is one. A suffix of "K", "M", "G", "T" or "P" may be used to multiply the number by 1024, 1048576, 1073741824, etc. respectively.

Cores
    Optionally specify the first thread CPU index numbers for the specific cores which can use this resource. For example, it may be strongly preferable to use specific cores with specific GRES devices (e.g. on a NUMA architecture). While Slurm can track and assign resources at the CPU or thread level, its scheduling algorithms used to co-allocate GRES devices with CPUs operates at a socket or NUMA level. Therefore it is not possible to preferentially assign GRES with different specific CPUs on the sane NUMA or socket and this option should be used to identify all cores on some socket Multiple cores may be specified using a comma delimited list or a range may be specified using a "-" separator (e.g. "0,1,2,3" or "0-3"). If the Cores configuration option is specified and a job is submitted with the --gres-flags=enforce-binding option then only the identified cores can be allocated with each generic resource; which will tend to improve performance of jobs, but slow the allocation of resources to them. If specified and a job is not submitted with the --gres-flags=enforce-binding option the identified cores will be preferred for scheduled with each generic resource. If --gres-flags=disable-binding is specified, then any core can be used with the resources, which also increases the speed of Slurm's scheduling algorithm but can degrade the application performance. The --gres-flags=disable-binding option is currently required to use more CPUs than are bound to a GRES (i.e. if a GPU is bound to the CPUs on one socket, but resources on more than one socket are required to run the job). If any core can be effectively used with the resources, then do not specify the cores option for improved speed in the Slurm scheduling logic. A restart of the slurmctld is needed for changes to the Cores option to take affect. NOTE: If your cores contain multiple threads only list the first thread of each core. The logic is such that it uses core instead of thread scheduling per GRES. Also note that since Slurm must be able to perform resource management on heterogeneous clusters having various core ID numbering schemes, an abstract index will be used instead of the physical core index. That abstract ID may not correspond to your physical core number. Basically Slurm starts numbering from 0 to n, being 0 the ID of the first processing unit (core or thread if hyperthreading is enabled) on the first socket, first core and maybe first thread, and then continuing sequentially to the next thread, core, and socket. The numbering generally coincides with the processing unit logical number (PU L#) seen in lstopo command output.

File
    Fully qualified pathname of the device files associated with a resource. The file name parsing logic includes support for simple regular expressions as shown in the example. This field is generally required if enforcement of generic resource allocations is to be supported (i.e. prevents a users from making use of resources allocated to a different user). If File is specified then Count must be either set to the number of file names specified or not set (the default value is the number of files specified). Slurm must track the utilization of each individual device If device file names are specified, which involves more overhead than just tracking the device counts. Use the File parameter only if the Count is not sufficient for tracking purposes. Except in the case of MPS support, if File is specified then Count must be either set to the number of file names specified or not set (the default value is the number of files specified). In the case of MPS support, each GPU would be identifed by name using the File parameter and Count would specify the number of MPS entries that would correspond to that GPU; typically 100 or some multiple of 100. NOTE: If you specify the File parameter for a resource on some node, the option must be specified on all nodes and Slurm will track the assignment of each specific resource on each node. Otherwise Slurm will only track a count of allocated resources rather than the state of each individual device file. NOTE: Drain a node before changing the count of records with File parameters (i.e. if you want to add or remove GPUs from a node's configuration). Failure to do so will result in any job using those GRES being aborted.

Links
    A comma-delimited list of numbers identifying the number of connections between this device and other devices to allow coscheduling of better connected devices. This is an ordered list in which the number of connections this specific device has to device number 0 would be in the first position, the number of connections it has to device number 1 in the second position, etc. A -1 indicates the device itself and a 0 indicates no connection. If specified, then this line can only contain a single GRES device (i.e. can only contain a single file via File). This is an optional value and is typically automatically determined using the NVIDIA NVML library. A typical use case would be to identify GPUs having NVLink connectivity. Note that for GPUs, the minor number assigned by the OS and used in the device file (i.e. the X in /dev/nvidiaX) is not necessarily the same as the device number/index. The device number is created by sorting the GPUs by PCI bus ID and then numbering them starting from the smallest bus ID.

Name
    Name of the generic resource. Any desired name may be used. Each generic resource has an optional plugin which can provide resource-specific options. Generic resources that currently include an optional plugin are:

    gpu Graphics Processing Unit

    nic Network Interface Card

    mic Intel Many Integrated Core (MIC) processor

NodeName
    An optional NodeName specification can be used to permit one gres.conf file to be used for all compute nodes in a cluster by specifying the node(s) that each line should apply to. The NodeName specification can use a Slurm hostlist specification as shown in the example below.

Type
    An arbitrary string identifying the type of device. For example, a particular model of GPU. If Type is specified, then Count is limited in size (currently 1024).

Ignore
    Defaults to false or no. If set to true or yes, this GRES will be ignored (for GRESs that support it). Currently, the only supported GRESs are Name=gpu. If all GRESs of the same name contain lines with Ignore=true, then all system-detected GRESs will be used except for the explicitly ignored GRESs. Specifying a NodeName will limit the scope of the ignored GRES to just that node.

EXAMPLES

##################################################################
# Slurm's Generic Resource (GRES) configuration file
## Define GPU devices with MPS support
##################################################################
Name=gpu Type=gtx560 File=/dev/nvidia0 COREs=0,1
Name=gpu Type=tesla File=/dev/nvidia1 COREs=2,3
Name=mps Count=100 File=/dev/nvidia0 COREs=0,1
Name=mps Count=100 File=/dev/nvidia1 COREs=2,3

##################################################################
# Slurm's Generic Resource (GRES) configuration file
## Ignore select GPU devices that are out of service on the current node
##################################################################
Name=gpu File=/dev/nvidia0 Ignore=true
Name=gpu File=/dev/nvidia1 Ignore=true
Name=gpu File=/dev/nvidia2 Ignore=true

##################################################################
# Slurm's Generic Resource (GRES) configuration file
## Overwrite system defaults and explicitly configure three GPUs
##################################################################
Name=gpu Type=tesla File=/dev/nvidia[0-1] COREs=0,1
# Name=gpu Type=tesla File=/dev/nvidia[2-3] COREs=2,3
# NOTE: nvidia2 device is out of service
Name=gpu Type=tesla File=/dev/nvidia3 COREs=2,3

##################################################################
# Slurm's Generic Resource (GRES) configuration file
## Use a single gres.conf file for all compute nodes - positive method
##################################################################
## Explicitly specify devices on nodes tux0-tux15
# NodeName=tux[0-15] Name=gpu File=/dev/nvidia[0-3]
# NOTE: tux3 nvidia1 device is out of service
NodeName=tux[0-2] Name=gpu File=/dev/nvidia[0-3]
NodeName=tux3 Name=gpu File=/dev/nvidia[0,2-3]
NodeName=tux[4-15] Name=gpu File=/dev/nvidia[0-3]

##################################################################
# Slurm's Generic Resource (GRES) configuration file
## Use a single gres.conf file for all compute nodes - negative method
##################################################################
## Ignore select system-detected devices on nodes tux0-tux15
NodeName=tux0 Name=gpu File=/dev/nvidia[1-2] Ignore=true
NodeName=tux[4,9,12] Name=gpu File=/dev/nvidia3 Ignore=true
NodeName=tux13 Name=gpu File=/dev/nvidia[0,4] Ignore=true
NodeName=tux15 Name=gpu File=/dev/nvidia0 Ignore=true

COPYING
Copyright (C) 2010 The Regents of the University of California. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
Copyright (C) 2010-2018 SchedMD LLC.

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

SEE ALSO

slurm.conf(5)