# Intel Knights Landing (KNL) User and Administrator Guide

## Overview

This document describes the unique features of Slurm on the computers with the Intel Knights Landing processor. You should be familiar with the Slurm's mode of operation on Linux clusters before studying the relatively few differences in Intel KNL system operation described in this document.

## User Tools

The desired NUMA and MCDRAM modes for a KNL processor should be specified using the -C or --constraints option of Slurm's job submission commands: salloc, sbatch, and srun. Currently available NUMA and MCDRAM modes are shown in the table below. Each node's available and current NUMA and MCDRAM modes are visible in the "available features" and "active features" fields respectively, which may be seen using the scontrol, sinfo, or sview commands. Note that a node may need to be rebooted to get the desired NUMA and MCDRAM modes and nodes may only be rebooted when they contain no running jobs (i.e. sufficient resources may be available to run a pending job, but until the node is idle and can be rebooted, the pending job may not be allocated resources). Also note that the job will be charged for resources from the time of resource allocation, which may include time to reboot a node into the desired NUMA and MCDRAM configuration.

While Slurm generally supports a very rich set of options for the node constraint options (exclusive OR, node counts for each constraint, etc.), only a simple AND operation is supported for KNL systems (specified using a comma or ampersand separator between specified constraints). Jobs may specify their desired NUMA and/or MCDRAM configuration. If no NUMA and/or MCDRAM configuration is specified, then a node with any possible value for that configuration will be used.

Type | Name | Description
--- | --- | ---
MCDRAM | cache | All of MCDRAM to be used as cache
MCDRAM | equal | MCDRAM to be used partly as cache and partly combined with primary memory
MCDRAM | flat | MCDRAM to be combined with primary memory into a "flat" memory space
NUMA | a2a | All to all
NUMA | hemi | Hemisphere
NUMA | snc2 | Sub-NUMA cluster 2
NUMA | snc4 | Sub-NUMA cluster 4
NUMA | quad | Quadrant

### Accounting

If a node requires rebooting for a job's required configuration, the job will be charged for the resource allocation from the time of allocation through the lifetime of the job, including the time consumed for booting the nodes. The job's time limit will be calculated from the time that all nodes are ready for use. For example, a job with a 10 minute time limit may be allocated resources at 10:00:00. If the nodes require rebooting, they might not be available for use until 10:20:00, 20 minutes after allocation, and the job will begin execution at that time. The job must complete no later than 10:30:00 in order to satisfy it's time limit (10 minutes after execution actually begins). However, the job will be charge for 30 minutes of resource use, which includes the boot time.

### Sample Use Cases

    $ sbatch -C flat,a2a -N2 --exclusive my.script
    $ srun --constraint=hemi,cache -n36 a.out
    $ srun --constraint=flat -n36 a.out

    $ sinfo -o "%30N %20b %f"
    NODELIST       ACTIVE_FEATURES  AVAIL_FEATURES
    nid000[10-11]
    nid000[12-35]  flat,a2a         flat,a2a,snc2,snc4,hemi,quad
    nid000[36-43]  cache,a2a        flat,equal,cache,a2a,snc2,snc4,hemi,quad

## System Administration

Three important components are required to use Slurm on an Intel KNL system.

1. The node features plugin manages the available and active features information available for each KNL node.
2. A configuration file is used to define various timeouts, default configuration, etc. The configuration file name and contents will depend upon the node features plugins used. See the knl.conf man page for more information.
3. A mechanism is required to boot nodes in the desired configuration. This mechanism must be integrated with existing Slurm infrastructure for rebooting nodes on user request (--reboot) plus power saving (powering down idle nodes and restarting them on demand).

In addition, there is a DebugFlags option of "NodeFeatures" which will generate detailed information about KNL operations.

Note that a node's KNL-specific available and active features are not included in the "slurm.conf" configuration file, but are set and the managed by the NodeFeatures plugin. Features which are not KNL-specific (e.g. rack number, "knl", etc.) will be copied from the node's "Features" configuration in "slurm.conf" to both the available and active feature fields and not modified by the NodeFeatures plugin.

### Mode of Operation

1. The node's configured "Features" are copied to the available and active feature fields.
2. The node features plugin determines the node's current MCDRAM and NUMA values as well as those which are available and adds those values to the node's active and available feature fields respectively. Note that these values may not be available until the node has booted and the slurmd daemon on the compute node sends that information to the slurmctld daemon.
3. Jobs will be allocated nodes already in the requested MCDRAM and NUMA mode if possible. If insufficient resources are available with the requested configuration then other nodes will be selected and booted into the desired configuration once no other jobs are active on the node. Until a node is idle, its configuration can not be changed. Note that node reboot time is roughly on the order of 20 minutes.

### Cray Configuration

On Cray systems NodeFeaturesPlugins should be set to "knl_cray".

The configuration file will be named "knl_cray.conf". The file will include the path to the capmc program (CapmcPath), which is used to get a node's available MCDRAM and NUMA modes, change the modes, power the node down, reboot it, etc. Note the "CapmcTimeout" parameter is the time required for the capmc program to respond to a request and NOT the time for a boot operation to complete.

Power saving mode is integrated with rebooting nodes in the desired mode. Programs named "capmc_resume" and "capmc_suspend" are provided to boot nodes in the desired mode. The programs are included in the main Slurm RPM and installed in the "sbin" directory. If powering down of idle nodes is not desired, then configure "ResumeProgram" in "slurm.conf" to the path of the "capmc_resume" file and configure "SuspendTime" to a huge value (e.g. "SuspendTime=30000000" will only power down a node which has been idle for about one year).

### Sample knl_cray.conf File

    # Sample knl_cray.conf
    CapmcPath=/opt/cray/capmc/default/bin/capmc
    CapmcTimeout=2000	# msec
    DefaultNUMA=a2a         # NUMA=all2all
    DefaultMCDRAM=cache     # MCDRAM=cache

### Sample slurm.conf File

    # Sample slurm.conf
    NodeFeaturesPlugins=knl_cray
    DebugFlags=NodeFeatures
    #
    ResumeProgram=/opt/slurm/default/sbin/capmc_resume
    SuspendTime=30000000
    ResumeTimeout=1800
    ...
    Nodename=default Sockets=1 CoresPerSocket=68 ThreadsPerCore=4 RealMemory=128000 Feature=knl
    NodeName=nid[00000-00127] State=UNKNOWN

Last modified 17 February 2016