# IBM Parallel Environment User and Administrator Guide

## Overview

This document describes the unique features of SLURM on the IBM computers with the Parallel Environment (PE) software. You should be familiar with the SLURM's mode of operation on Linux clusters before studying the relatively few differences in operation on systems with PE, which are described in this document.

## User Tools

The normal set of SLURM user tools: srun, scancel, sinfo, squeue, scontrol, etc. provide all of the expected services. The only SLURM command not supported is sattach. Job steps are launched using the srun command, which translates its options and invokes IBM's poe command. The poe command actually launches the tasks. The poe command may also be invoked directly if desired. The actual task launch process is as follows:

1. Invoke srun command with desired options.
2. The srun command creates a job allocation (if necessary).
3. The srun command translates its options and invokes the poe command.
4. The poe command loads a SLURM library that provides various resource management functionality.
5. The poe command, through the SLURM library, launches a process named "pmdv12" on the appropriate compute nodes. Note that the "v12" on the end of the process name represents the version number of the "pmd" process and is subject to change.
6. The poe command interacts with the pmdv12 process to launch the application tasks, handle their I/O, etc. Since the task launch procedure occurs outside of SLURM's control, none of the normal task-level SLURM support is available.
7. The poe command, through the SLURM library, notes the completion of the job step.

### Network Options

Each job step can specify it's desired network options. For example, one job step may use IP mode communications and the next use User Space (US) mode communications. Network specifications may be specified using srun's --network option or the SLURM_NETWORK environment variable. Supported network options include:

- Network protocol

- ip Internet protocol, version 4
- ipv4 Internet protocol, version 4 (default)
- ipv6 Internet protocol, version 6
- us User Space protocol, may be combined with ibv4 or ipv6

- Programming interface

- lapi Low-level Application Programming Interface
- mpi Message Passing Interface (default)
- pami Parallel Active Message Interface
- upc Unified Parallel C Interface

- Other options

- bulk_xfer [=resources] Enable bulk transfer of data using Remote Direct-Memory Access (RDMA). The optional resources specification is a numeric value which can have a suffix of "k", "K", "m", "M", "g" or "G" for kilobytes, megabytes or gigabytes. NOTE: The resources specification is not supported by the underlying IBM infrastructure as of Parallel Environment version 2.2 and no value should be specified at this time.
- cau=count Specify the count of Collective Acceleration Units (CAU) required per programming interface. Default value is zero. Applies only to IBM Power7-IH processors. POE requires that if cau has a non-zero value then devtype=IB or devtype=HFI must be explictly specified.
- devname=name Specify the name of an network adapter to use. For example: "eth0" or "mlx4_0".
- devtype=type Specify the device type to use for communications. The supported values of type are: "IB" (InfiniBand), "HFI" (P7 Host Fabric Interface), "IPONLY" (IP-Only interfaces), "HPCE" (HPC Ethernet), and "KMUX" (Kernel Emulation of HPCE). The devices allocated to a job must all be of the same type. The default value depends upon depends upon what hardware is available and in order of preferences is IPONLY (which is not considered in User Space mode), HFI, IB, HPCE, and KMUX.
- immed=count Specify the count of immediate send slots per adapter window. Default value is zero. Applies only to IBM Power7-IH processors.
- instances=count Specify number of network connections for each task on each network connection. The default instance count is 1.
- sn_all Use all available switch adapters (default). This option can not be combined with sn_single.
- sn_single Use only one switch adapters. This option can not be combined with sn_all. If multiple adapters of different types exist, the devname and/or devtype option can also be used to select one of them.

Examples of network option use:
--network=sn_all,mpi
Allocate one switch window per task on each node and every network supporting MPI.
--network=sn_all,mpi,bulk_xfer,us
Allocate one switch window per task on each node and every network supporting MPI and user space communications. Reserve resources for RDMA.
--network=sn_all,instances=3,mpi
Allocate three switch window per task on each node and every network supporting MPI.
--network=sn_all,mpi,pami
Allocate one switch window per task on each node and every network supporting MPI and a second window supporting PAMI.
--network=devtype=ib,instances=2,lapi,mpi
On every Infiniband network connection, allocate two switch windows each for both lapi and mpi interfaces. If each node has one Infinband network connection, this would result in four switch windows per task.

NOTE: Switch resources on a node are shared between all job steps on that node. If a job step can not be initiated due to insufficient switch resources being available, that job step will periodically retry getting resources for the lifetime of the job unless srun's --immediate option is used.

### Debugging

Most debuggers require detailed information about launched tasks such as host name, process ID, etc. Since that information is only available from poe (which launches those tasks), the srun command wrapper can not be used for most debugging purposes. You or the debugging tool must invoke the poe command directly. In order to facilitate use of poe, srun's
--launch-cmd option may be used with the options normally used. srun will then print the equivalent poe command line, which can subsequently be used with the debugger.

### Checkpoint

FIXME: We need to validate all of this SLURM supports checkpoint via poe. In order to enable checkpoint, the shell executing the poe command must itself be initiated with the environment variable CHECKPOINT=yes. One file is written for each node on which the job is executing, plus another for the script executing poe.a By default, the checkpoint files will be written to the current working directory of the job. Names and locations of these files can be controlled using the environment variables MP_CKPTFILE and MP_CKPTDIR. Use the squeue command to identify the job and job step of interest. To initiate a checkpoint in which the job step will continue execution, use the command:
scontrol check create job_id.step_id
To initiate a checkpoint in which the job step will terminate afterwards, use the command:
scontrol check vacate job_id.step_id

### Unsupported Options

Some SLURM options can not be supported by PE and the following srun options are silently ignored:

- -D, --chdir (set working directory)
- -K, --kill-on-bad-exit (terminate step if any task has a non-zero exit code)
- -k, --no-kill (set to not kill job upon node failure)
- --ntasks-per-core (number of tasks to invoke per code)
- --ntasks-per-socket (number of tasks to invoke per socket)
- -O, --overcommit (over subscribe resources)
- --resv-ports (communication ports reserved for OpenMPI)
- --runjob-opts (used only on IBM BlueGene/Q systems)
- --signal (signal to send when near time limit and the remaining time required)
- --sockets-per-node (number of sockets per node required)
- --task-epilog (per-task epilog program)
- --task-prolog (per-task prolog program)
- -u, --unbuffered (avoid line buffering)
- -W, --wait (specify job swait time after first task exit)
- -Z, --no-allocate (launch tasks without creating job allocation>

In addition, file name specifications with expression substitution (e.g. file names including "%j" for job_ID, "%J" for job_ID.step_ID, "%s" for step_ID "%t" for task_ID or "%n" for node_ID) are not supported. This effects the following options:

- -e, --error
- -i, --input
- -o, --output

The -L/--label option differs slighly in that when the output from multiple tasks are identical, they are combined on a single line with the prefix identifying which task(s) generated the output. In addition, there is a colon but no space between the task IDs and output. For example:

    # SLURM OUTPUT
    0: foo
    1: foo
    0: bar
    1: barr

    # POE OUTPUT
    0-1:foo
    0:bar
    1:barr

### Environment Variables

Since SLURM is not directly launching user tasks, the following environment variables are NOT available with POE:

- SLURM_CPU_BIND_LIST
- SLURM_CPU_BIND_TYPE
- SLURM_CPU_BIND_VERBOSE
- SLURM_CPUS_ON_NODE
- SLURM_GTIDS
- SLURM_LAUNCH_NODE_IPADDR
- SLURM_LOCALID
- SLURM_MEM_BIND_LIST
- SLURM_MEM_BIND_TYPE
- SLURM_MEM_BIND_VERBOSE
- SLURM_NODEID
- SLURM_PROCID
- SLURM_SRUN_COMM_HOST
- SLURM_SRUN_COMM_PORT
- SLURM_TASK_PID
- SLURM_TASKS_PER_NODE
- SLURM_TOPOLOGY_ADDR
- SLURM_TOPOLOGY_ADDR_PATTERN

Note that POE sets a variety of environment variables that provide similar information to some the missing SLURM environment variables. Particularly node the following variables:

- MP_I_UPMD_HOSTNAME (local hostname)
- MP_CHILD (global task ID)

### Gang Scheduling

SLURM can be configured to gang schedule (time slice) parallel jobs by alternately suspending and resuming them. Depending upon the the number of jobs configured to time slice and the time slice interval (as specified in the slurm.conf file using the Shared and SchedulerTimeSlice options), the job may experience communication timeouts. Set the environment variable MP_TIMEOUT to specify an appropriate communication timeout value. Note that the default timeout is 150 seconds. See Gang Scheduling for more information.

    export MP_TIMEOUT=600

### Other User Notes

If the srun command's --label option is used, note that POE's prefix is the task ID followed by a colon. The prefix with native SLURM is the task ID followed by a colon followed by one space.

POE forwards a limited set of signals to spawned tasks, mostly signals that indicate a fatal error. It does not forward SIGUSR1 or SIGUSR2 signals.

## System Administration

There are several critical SLURM configuration parameters for use with PE. These configuration parameters should be set in your slurm.conf file. LaunchType defines the task launch mechanism to be used and must be set to launch/poe. SwitchType defines the mechanism used to manage the network switch and it must be set to switch/nrt and use IBM's Network Resource Table (NRT) interface. Task launch is slower in this environment than with a typical Linux cluster and the MessageTimeout must be configured to a sufficiently large value so that large parallel jobs can be launched without SLURM's job step credentials expiring. When switch resources are allocated to a job, all processes spawned by that job must be terminated before the switch resources can be released for use by another program. This means that reliable tracking of all spawned processes is critical for switch use. Use of ProctrackType=proctrack/cgroup is strongly recommended. Use of any other process tracking plugin significantly increases the likelyhood of orphan processes that must be manually identified and killed in order to reuse switch resources.

    # Excerpt of slurm.conf
    LaunchType=launch/poe
    SwitchType=switch/nrt
    MessageTimeout=30
    ProctrackType=proctrack/cgroup

In order for these plugins to be built, the locations of the POE Resource Manager header file (permapi.h) the NRT header file (nrt.h) and NRT library (libnrt.so) must be identified at the time the SLURM is built. SLURM searches for the header files in the /usr/include directory by default. If the files are not installed there, you can specify a different location using the --with-nrth=PATH option to the configure program, where "PATH" is the fully qualified pathname of the parent directory of the nrt.h and permapi.h files. SLURM searches for the libnrt.so file in the /usr/lib and /usr/lib64 directories by default. If the file is not installed there, you can specify a different location using the --with-nrtlib=PATH option to the configure program, where "PATH" is the fully qualified pathname of the parent directory of the libnrt.so file. Alternately these values may be specified in your ~/.rpmmacros file. For example:

    %_with_nrth      "/opt/ibmhpc/pecurrent/base/include"
    %_with_libnrt    "/opt/ibmhpc/pecurrent/base/intel/lib64"

The poe command interacts with SLURM by loading a SLURM library providing a variety of functions for its use. You must specify the location of that library and note that SLURM is the resource manager in the file named "/etc/poe.limits". The library name is "libpermapi.so" and it is in installed with the other SLURM libraries in the subdirectory "lib/slurm". A sample "/etc/poe.limits" file is shown below. You will need to modify the value of MP_PE_RMLIB to match SLURM's installation location on your system.
NOTE: The poe command is loading and using the libpermapi.so library initially from the /usr/lib64 directory. It later reads the /etc/poe.limits file and loads the correct library. In order for poe to work with SLURM it needs to use the "libpermapi.so" generated by SLURM for all of its functions. Until poe is modified to only load the correct library, it is necessary for the file /usr/lib64/libpermapi.so to contain SLURM's library or a link to it.

    #
    # Sample /etc/poe.limits
    # Modify the path below as appropriate
    #
    MP_PE_RMLIB=/usr/local/lib/slurm/libpermapi.so

### Job Scheduling

SLURM can be configured to gang schedule (time slice) parallel jobs by alternately suspending and resuming them. Depending upon the the number of jobs configured to time slice and the time slice interval (as specified in the slurm.conf file using the Shared and SchedulerTimeSlice options), the job may experience communication timeouts. Set the environment variable MP_TIMEOUT to specify an appropriate communication timeout value. Note that the default timeout is 150 seconds. See Gang Scheduling for more information.

    export MP_TIMEOUT=600

SLURM also can support long term job preemption with IBM's Parallel Environment. Job's can be explicitly preempted and later resumed using the scontrol suspend <jobid> and scontrol resume <jobid> commands. This functionality relies upon NRT functions to suspend/resume programs and reset MPI timeouts. Note that SLURM suports the preemption only of whole jobs rather than individual job steps. A suspended job will relinquish CPU resources, but retain memory and switch window resources. Note that the long term suspension of jobs with any allocated Collective Acceleration Units (CAU) is disabled and an error message to that effect will be generated in response to such a request. In addition, verion 1200 or higher of IBM's NRT API is required to support this functionality.

### Design Notes

It is possible to configure SLURM and LoadLeveler to simultaneously exist on a cluster, however each scheduler must be configured to manage different compute nodes (e.g. LoadLeveler can manage compute nodes "tux[1-8]" and SLURM can manaage compute nodes "tux[9-16]" on the same cluster).

The srun command uses the launch/poe plugin to launch the poe program. Then poe uses the launch/slurm plugin to launch the "pmd" process on the compute nodes, so two launch plugins are actually used.

Depending upon job size and network options, allocating and deallocating switch resources can take multple seconds per node and the process of launching applications on multiple nodes is not well parallelized. This is outside of SLURM's control.

### Debugging Notes

It is possible to generate detailed logging of all switch/nrt actions and data by configuring DebugFlags=switch.

The envirnoment variable MP_INFOLEVEL can be used to enable the logging of POE debug messages. To enable fairly detailed logging, set MP_INFOLEVEL=6.

The Protocol Network Services Daemon (PNSD) manages the Network Resource Table (NRT) information on each node. It's logs are written to the file /etc/serverlog, which may be useful to diagnose problems. In order to execute PNSD in debug node (for extra debugging information), run the following commands as user root:

    stopsrc -s pnsd
    startsrc -s pnsd -a -D

Last modified 16 August 2012