# Intel Knights Landing (KNL) User and Administrator Guide

## Overview

This document describes the unique features of Slurm on the computers with the Intel Knights Landing processor. You should be familiar with the Slurm's mode of operation on Linux clusters before studying the relatively few differences in Intel KNL system operation described in this document.

## User Tools

The desired NUMA and MCDRAM modes for a KNL processor should be specified using the -C or --constraints option of Slurm's job submission commands: salloc, sbatch, and srun. Currently available NUMA and MCDRAM modes are shown in the table below. Each node's available and current NUMA and MCDRAM modes are visible in the "available features" and "active features" fields respectively, which may be seen using the scontrol, sinfo, or sview commands. Note that a node may need to be rebooted to get the desired NUMA and MCDRAM modes and nodes may only be rebooted when they contain no running jobs (i.e. sufficient resources may be available to run a pending job, but until the node is idle and can be rebooted, the pending job may not be allocated resources). Also note that the job will be charged for resources from the time of resource allocation, which may include time to reboot a node into the desired NUMA and MCDRAM configuration. While Slurm generally supports a very rich set of options for the node constraint options (exclusive OR, node counts for each constraint, etc.), only a simple AND operation is supported for KNL systems (specified using a comma or ampersand separator between specified constraints).

Type | Name | Description
--- | --- | ---
MCDRAM | cache | All of MCDRAM to be used as cache
MCDRAM | equal | MCDRAM to be used partly as cache and partly combined with primary memory
MCDRAM | flat | MCDRAM to be combined with primary memory into a "flat" memory space
NUMA | a2a | All to all
NUMA | hemi | Hemisphere
NUMA | snc2 | Sub-NUMA cluster 2
NUMA | snc4 | Sub-NUMA cluster 4
NUMA | quad | Quadrant

### Sample Use Cases

    $ sbatch -C flat,a2a -N2 --exclusive my.script
    $ srun --constraint=hemi,cache -n36 a.out

    $ sinfo -o "%30N %20b %f"
    NODELIST                      ACTIVE_FEATURES  AVAIL_FEATURES
    nid[00010-00011,00016-00021]
    nid[00012-00035]              flat,a2a         flat,a2a,snc2,snc4,hemi,quad
    nid[00036-00043]              flat,a2a         flat,equal,cache,a2a,snc2,snc4,hemi,quad

## System Administration

Two unique components are required to use Slurm on an Intel KNL system.

1. The NodeFeatures plugin manages the available and active features available for each KNL node. On Cray systems NodeFeaturesPlugins should be set to "knl_cray".
2. A configuration file is used to define various timeouts, default configuration, etc. The configuration file name and contents will depend upon the NodeFeatures used. On Cray systems the file will be named "knl_cray.conf". See the knl.conf man page for more information.
3. BOOTING

In addition, there is a DebugFlags option of "NodeFeatures" which will generate detailed information about KNL operations.

### Sample knl_cray.conf File

    # knl_cray.conf
    CapmcPath=/opt/cray/capmc/default/bin/capmc
    CapmcTimeout=600	# msec
    DefaultNUMA=a2a         # NUMA=all2all
    DefaultMCDRAM=cache     # MCDRAM=cache

### Sample slurm.conf File

    # Select portions of slurm.conf
    NodeFeaturesPlugins=knl_cray
    DebugFlags=NodeFeatures
    ..
    Nodename=default feature=knl Sockets=1 CoresPerSocket=68 ThreadsPerCore=4 RealMemory=128000
    NodeName=nid[00000-00127] State=UNKNOWN

When the slurmctld daemon starts, the configued NodeFeatures plugin will load available and active NUMA and MCDRAM modes on the KNL node. It will then add these values to any node Features for the node defined in slurm.conf. Separate active and available features for each node can be viewed with the scontrol, sinfo, and sview commands.

Last modified 4 February 2016