# Intel Knights Landing (KNL) User and Administrator Guide

## Overview

This document describes the unique features of Slurm on the computers with the Intel Knights Landing processor. You should be familiar with the Slurm's mode of operation on Linux clusters before studying the relatively few differences in Intel KNL system operation described in this document.

## User Tools

The desired NUMA and MCDRAM modes for a KNL processor should be specified using the -C or --constraints option of Slurm's job submission commands: salloc, sbatch, and srun. Currently available NUMA and MCDRAM modes are shown in the table below. Each node's available and current NUMA and MCDRAM modes are visible in the "available features" and "active features" fields respectively, which may be seen using the scontrol, sinfo, or sview commands. Note that a node may need to be rebooted to get the desired NUMA and MCDRAM modes and nodes may only be rebooted when they contain no running jobs (i.e. sufficient resources may be available to run a pending job, but until the node is idle and can be rebooted, the pending job may not be allocated resources). Also note that the job will be charged for resources from the time of resource allocation, which may include time to reboot a node into the desired NUMA and MCDRAM configuration.

While Slurm generally supports a very rich set of options for the node constraint options (exclusive OR, node counts for each constraint, etc.), only a simple AND operation is supported for KNL systems (specified using a comma or ampersand separator between specified constraints). Jobs may specify their desired NUMA and/or MCDRAM configuration. If no NUMA and/or MCDRAM configuration is specified, then a node with any possible value for that configuration will be used.

Type | Name | Description
--- | --- | ---
MCDRAM | cache | All of MCDRAM to be used as cache
MCDRAM | equal | MCDRAM to be used partly as cache and partly combined with primary memory
MCDRAM | flat | MCDRAM to be combined with primary memory into a "flat" memory space
NUMA | a2a | All to all
NUMA | hemi | Hemisphere
NUMA | snc2 | Sub-NUMA cluster 2
NUMA | snc4 | Sub-NUMA cluster 4
NUMA | quad | Quadrant

Jobs requiring some or all of the KNL high bandwidth memory (HBM) should explicitly request that memory using Slurm's Generic RESource (GRES) options. The HBM will always be known by Slurm GRES name of "hbm". Examples below demonstrate use of HBM.

### Accounting

If a node requires rebooting for a job's required configuration, the job will be charged for the resource allocation from the time of allocation through the lifetime of the job, including the time consumed for booting the nodes. The job's time limit will be calculated from the time that all nodes are ready for use. For example, a job with a 10 minute time limit may be allocated resources at 10:00:00. If the nodes require rebooting, they might not be available for use until 10:20:00, 20 minutes after allocation, and the job will begin execution at that time. The job must complete no later than 10:30:00 in order to satisfy it's time limit (10 minutes after execution actually begins). However, the job will be charge for 30 minutes of resource use, which includes the boot time.

### Sample Use Cases

    $ sbatch -C flat,a2a -N2 -gres=hbm:8g --exclusive my.script
    $ srun --constraint=hemi,cache -n36 a.out
    $ srun --constraint=flat --gres=hbm:2g -n36 a.out

    $ sinfo -o "%30N %20b %f"
    NODELIST       ACTIVE_FEATURES  AVAIL_FEATURES
    nid000[10-11]
    nid000[12-35]  flat,a2a         flat,a2a,snc2,snc4,hemi,quad
    nid000[36-43]  cache,a2a        flat,equal,cache,a2a,hemi,quad

### Network Topology

Slurm will optimize performance using those resources available without rebooting. If node rebooting is required, then it will optimize layout with respect to network bandwidth using both nodes currently in the desired configuration and those which can be made available after rebooting. This can result in more nodes being rebooted than strictly needed, but will improve application performance.

Users can specify they want all resources allocated on a specific count of leaf switches (Dragonfly group) using Slurm's --switches option. They can also specify how much additional time they are willing to wait for such a configuration. If the desired configuration can not be made available within the specified time interval, the job will be allocated nodes optimized with respect to network bandwidth to the extent possible. On a Dragonfly network, this means allocating resources over either single group or distributed evenly over as many groups as possible. For example:

    srun --switches=1@10:00 N16 a.out

Note that system administrators can disable use of the --switches option or limit the amount of time the job can be deferred using the SchedulerParameters max-switch-wait option.

### Booting Problems

If node boots fail, those nodes are drained and the job is requeued so that it can be allocated a different set of nodes. The nodes originally allocated to the job will remain available to the job, so likely a small number of additional nodes will be required.

## System Administration

Three important components are required to use Slurm on an Intel KNL system.

1. The node features plugin manages the available and active features information available for each KNL node.
2. A configuration file is used to define various timeouts, default configuration, etc. The configuration file name and contents will depend upon the node features plugins used. See the knl.conf man page for more information.
3. A mechanism is required to boot nodes in the desired configuration. This mechanism must be integrated with existing Slurm infrastructure for rebooting nodes on user request (--reboot) plus power saving (powering down idle nodes and restarting them on demand).

In addition, there is a DebugFlags option of "NodeFeatures" which will generate detailed information about KNL operations.

Note that a node's KNL-specific available and active features are not included in the "slurm.conf" configuration file, but are set and the managed by the NodeFeatures plugin. Features which are not KNL-specific (e.g. rack number, "knl", etc.) will be copied from the node's "Features" configuration in "slurm.conf" to both the available and active feature fields and not modified by the NodeFeatures plugin.

Slurm version 16.05 does not support the concept of multiple NUMA nodes within a single socket. If a KNL node is booted with multiple NUMA, then each NUMA node will appear in Slurm as a separate socket. In the slurm.conf configuration file, set "FastSchedule=1" and node socket and core counts to values which are appropriate for some NUMA mode to be used on the node. When the node boots and the slurmd daemon on the node starts, it will report to the slurmctld daemon the node's actual socket (NUMA) and core counts, which will update Slurm data structures for the node to the values which are currently configured. Note that Slurm (version 16.05) current does not support the concept of differing numbers of cores in each socket (or NUMA node). We are currently working to address these issues.

### Mode of Operation

1. The node's configured "Features" are copied to the available and active feature fields.
2. The node features plugin determines the node's current MCDRAM and NUMA values as well as those which are available and adds those values to the node's active and available feature fields respectively. Note that these values may not be available until the node has booted and the slurmd daemon on the compute node sends that information to the slurmctld daemon.
3. Jobs will be allocated nodes already in the requested MCDRAM and NUMA mode if possible. If insufficient resources are available with the requested configuration then other nodes will be selected and booted into the desired configuration once no other jobs are active on the node. Until a node is idle, its configuration can not be changed. Note that node reboot time is roughly on the order of 20 minutes.

### Cray Configuration

On Cray systems NodeFeaturesPlugins should be set to "knl_cray".

The configuration file will be named "knl_cray.conf". The file will include the path to the capmc program (CapmcPath), which is used to get a node's available MCDRAM and NUMA modes, change the modes, power the node down, reboot it, etc. Note the "CapmcTimeout" parameter is the time required for the capmc program to respond to a request and NOT the time for a boot operation to complete.

Power saving mode is integrated with rebooting nodes in the desired mode. Programs named "capmc_resume" and "capmc_suspend" are provided to boot nodes in the desired mode. The programs are included in the main Slurm RPM and installed in the "sbin" directory and must be installed on the Cray "sdb" node. If powering down of idle nodes is not desired, then configure "ResumeProgram" in "slurm.conf" to the path of the "capmc_resume" file and configure "SuspendTime" to a huge value (e.g. "SuspendTime=30000000" will only power down a node which has been idle for about one year).

The GresTypes configuration parameter should include "hbm" to identify High Bandwidth Memory (HBM) as being a consumable resources on compute nodes. Additional GresTypes can be specified as needed in a comma separated list. The amount of HBM on each node should not be configured in a Slurm configuration file, but that information will be loaded by the knl_cray plugin using information provided by the capmc program.

### Sample knl_cray.conf File

    # Sample knl_cray.conf
    CapmcPath=/opt/cray/capmc/default/bin/capmc
    CapmcTimeout=2000	# msec
    DefaultNUMA=a2a         # NUMA=all2all
    DefaultMCDRAM=cache     # MCDRAM=cache

### Sample slurm.conf File

    # Sample slurm.conf
    NodeFeaturesPlugins=knl_cray
    DebugFlags=NodeFeatures
    GresTypes=hbm
    #
    ResumeProgram=/opt/slurm/default/sbin/capmc_resume
    SuspendTime=30000000
    ResumeTimeout=1800
    ...
    Nodename=default Sockets=1 CoresPerSocket=68 ThreadsPerCore=4 RealMemory=128000 Feature=knl
    NodeName=nid[00000-00127] State=UNKNOWN

Last modified 30 March 2016