# SLURM User and Administrator Guide for Cray systems

## User Guide

This document describes the unique features of SLURM on Cray computers. You should be familiar with the SLURM's mode of operation on Linux clusters before studying the differences in Cray system operation described in this document.

SLURM version 2.3 is designed to operate as a job scheduler over Cray's Application Level Placement Scheduler (ALPS). Use SLURM's sbatch or salloc commands to create a resource allocation in ALPS. Then use ALPS' aprun command to launch parallel jobs within the resource allocation. The resource allocation is terminated once the the batch script or the salloc command terminate. Alternately there is an aprun wrapper distributed with SLURM in contribs/cray/srun2aprun which will translate srun options into the equivalent aprun options. This wrapper will also execute salloc as needed to create a job allocation in which to run the aprun command. The srun2aprun script contains two new options: --man will print a summary of the options including notes about which srun options are not supported and --alps=" which can be used to specify aprun options which lack an equivalent within srun. For example, srun2aprun --alps="-a xt" -n 4 a.out. Since aprun is used to launch tasks (the equivalent of a SLURM job step), the job steps will not be visible using SLURM commands. Other than SLURM's srun command being replaced by aprun and the job steps not being visible, all other SLURM commands will operate as expected.

SLURM node names will be of the form "nid#####" where "#####" is a five digit sequence number. Other information available about the node is it's XYZ coordinate in the node's NodeAddr field and it's component label in the HostNodeName field. The format of the component lable is "c#-#c#s#n#" where the "#" fields represent in order: cabinet, row, cate, blade or slot, and node. For example "c0-1c2s5n3" is cabinet 0, row 1, cage 3, slot 5 and node 3.

## Administrator Guide

### Install supporting rpms

The build requires a few -devel RPMs listed below. You can obtain these from SuSe/Novel.

- CLE 2.x uses SuSe SLES 10 packages (rpms may be on the normal isos)
- CLE 3.x uses Suse SLES 11 packages (rpms are on the SDK isos, there are two SDK iso files for SDK)

You can check by

- Going onto the bootnode
- xtopview
- rpm -qa

The list of packages that should be installed is:

- expat-2.0.xxx
- libexpat-devel-2.0.xxx
- cray-MySQL-devel-enterprise-5.0.64 (this should be on the Cray iso)

### Create a build root

The build is done on a normal service node, where you like (e.g. /ufs/slurm/build would work). Most scripts check for the environment variable LIBROOT. You can either edit the scripts or export this variable. Easiest way:

- export LIBROOT=/ufs/slurm/build
- mkdir -vp $LIBROOT
- cd $LIBROOT

### Install modulefile

This file is distributed as part the SLURM tar-ball in contribs/cray/opt_modulefiles_slurm. Install it as /opt/modulefiles/slurm (or anywhere else in your module path). It means that you can use Munge as soon as it is built.

### Build and install Munge

Munge is the authentication daemon and needed by SLURM.

- cp munge_build_script.sh $LIBROOT
- mkdir -vp ${LIBROOT}/munge/zip
- Download munge-0.5.9.tar.bz2 or newer from http://code.google.com/p/munge/downloads/list
- Copy that into ${LIBROOT}/munge/zip
- Execute ./munge_build_script.sh

This generates a tar-ball called $LIBROOT/munge_build-.*YYYY-MM-DD.tar.gz which has already the correct paths - in particular, the libmunge.* need to reside in /usr/lib64 since SLURM plugins call them.

Install the tar-ball by executing:

- scp $LIBROOT/munge_build-.*YYYY-MM-DD.tar.gz boot:
- ssh boot
- tar -zxvf munge_build-*.tar.gz -C /rr/current

### Configure Munge

The following steps apply to each service node where

- The slurmd or slurmctld daemon will run and/or
- Users will be submitting jobs

1. On each login host create directory hierarchy:
   mkdir --mode=0711 -vp /var/lib/munge
   mkdir --mode=0700 -vp /var/log/munge
   mkdir --mode=0755 -vp /var/run/munge
2. module load slurm as root, to get the correct SLURM paths
3. munged --key-file /opt/slurm/munge/etc/munge.key
4. Try a "munge -n" to see if munged accepts input

When done, verify network connectivity by executing:

- munge -n |ssh other-login-host /opt/slurm/munge/bin/unmunge

If you decide to keep the installation, you may be interested in automating the process using an init.d script distributed with the Munge as src/etc/munge.init. This is installed by

- cp munge.init /rr/current/software/munge
- xtopview -c login
- cp /software/munge /etc/init.d
- chkconfig munge on

### Build and Configure SLURM

SLURM can be built and installed as on any other computer as described Quick Start Administrator Guide. When building SLURM's slurm.conf configuration files, use the NodeName parameter to specify all batch nodes to be scheduled. Note that the NodeAddr and NodeHostName fields should not be configured, but will be set by SLURM using data from ALPS. NodeAddr be set to the node's XYZ coordinate and be used by SLURM's smap and sview commands. NodeHostName will be set to the node's component label. The format of the component label is "c#-#c#s#n#" where the "#" fields represent in order: cabinet, row, cate, blade or slot, and node. For example "c0-1c2s5n3" is cabinet 0, row 1, cage 3, slot 5 and node 3.

The slurmd daemons will not execute on the compute nodes, but will execute on one or more front end nodes. It is from here that batch scripts will execute aprun commands to launch tasks. This is specified in the slurm.conf file by using the FrontendName and optionally the FrontEndAddr fields as seen in the examples below.

You need to specify the appropriate resource selection plugin (the SelectType option in SLURM's slurm.conf configuration file). Configure SelectType to select/cray The select/cray plugin provides an interface to ALPS plus issues calls to the select/linear, which selects resources for jobs using a best-fit algorithm to allocate whole nodes to jobs (rather than individual sockets, cores or threads).

The last Cray-specific configuration parameter controls the system topology information used to optimize resource allocations and must be configured as TopoType=topology/node_rank. The topology/node_rank plugin orders the nodes such that nodes which are close in three dimensional space a put close in one dimensional space. Since SLURM uses a best-fit algorithm in one dimensional space, the resulting job allocations should achieve good locality in three dimensional space also. For more information about how this is achieved, see SLURM's Topology Guide.

    # slurm.conf for Cray XT system of with 512 nodes with the first 80 nodes
    # reserved for login, service, interactive, etc. use
    # Parameters removed here
    SelectType=select/cray
    TopoType=topology/node_rank
    FrontendName=front_end[00-03]
    NodeName=nid[00080-00511] CPUs=8 RealMemory=2048
    PartitionName=batch Nodes=nid[00080-00511] Default=Yes State=UP

SLURM supports an optional cray.conf file containing Cray-specific configuration parameters. This file must be located in the same directory as the slurm.conf file.

AlpsDir
  Fully qualified pathname of the directory in which ALPS is installed. The default value is /usr.
apbasil
  Fully qualified pathname to the apbasil command. The default value is /usr/bin/apbasil.
apkill
  Fully qualified pathname to the apkill command. The default value is /usr/bin/apkill.
SDBdb
  Name of the ALPS database. The default value is XTAdmin.
SDBhost
  Hostname of the database server. The default value is sdb.
SDBpass
  Password used to access the ALPS database. The default value is NULL.
SDBport
  Port used to access the ALPS database. The default value is 0.
SDBuser
  Name of user used to access the ALPS database. The default value is NULL.

    # Example cray.conf file
    apbasil=/opt/alps_simulator_40_r6768/apbasil.sh
    SDBhost=localhost
    SDBuser=alps_user
    SDBdb=XT5istanbul

One additional configuration script can be used to insure that the slurmd daemons execute with the highest resource limits possible, overriding default limits on Suse systems. Depending upon what resource limits are propagated from the user's environment, lower limits may apply to user jobs, but this script will insure that higher limits are possible. Copy the file contribs/cray/etc_sysconfig_slurm into /etc/sysconfig/slurm for these limits to take effect. This script is executed from /etc/init.d/slurm, which is typically executed to start the SLURM daemons. An excerpt of contribs/cray/etc_sysconfig_slurmis shown below.

    #
    # /etc/sysconfig/slurm for Cray XT/XE systems
    #
    # Cray is SuSe-based, which means that ulimits from
    # /etc/security/limits.conf will get picked up any time SLURM is
    # restarted e.g. via pdsh/ssh. Since SLURM respects configured limits,
    # this can mean that for instance batch jobs get killed as a result
    # of configuring CPU time limits. Set sane start limits here.
    #
    # Values were taken from pam-1.1.2 Debian package
    ulimit -t unlimited	# max amount of CPU time in seconds
    ulimit -d unlimited	# max size of a process's data segment in KB

Copyright (C) 2011 SchedMD LLC.
Supported by the Oak Ridge National Laboratory Extreme Scale Systems Center

Last modified 6 April 2011