# SLURM User and Administrator Guide for Cray systems

NOTE: As of 20 February 2010, the SLURM interface to Cray systems is still under development.

## User Guide

This document describes the unique features of SLURM on Cray computers. You should be familiar with the SLURM's mode of operation on Linux clusters before studying the differences in Cray system operation described in this document.

SLURM version 2.3 is designed to operate as a job scheduler over Cray's Application Level Placement Scheduler (ALPS). Use SLURM's sbatch or salloc commands to create a resource allocation in ALPS. Then use ALPS' aprun command to launch parallel jobs within the resource allocation. The resource allocation is terminated once the the batch script or the salloc command terminate. Other than SLURM's srun command being replaced by aprun, all other SLURM commands will operate as expected.

SLURM node names will be of the form "nid#####" where "#####" is a five digit sequence number. Other information available about the node is it's XYZ coordinate in the node's NodeAddr field and it's component label in the HostNodeName field. The format of the component lable is "c#-#c#s#n#" where the "#" fields represent in order: cabinet, row, cate, blade or slot, and node. For example "c0-1c2s5n3" is cabinet 0, row 1, cage 3, slot 5 and node 3.

## Administrator Guide

### Install supporting rpms

The build requires a few -devel RPMs listed below. You can obtain these from SuSe/Novel.

- CLE 2.x uses SuSe SLES 10 packages (rpms may be on the normal isos)
- CLE 3.x uses Suse SLES 11 packages (rpms are on the SDK isos, there are two SDK iso files for SDK)

You can check by

- Going onto the bootnode
- xtopview
- rpm -qa

The list of packages that should be installed is:

- expat-2.0.xxx
- libexpat-devel-2.0.xxx
- cray-MySQL-devel-enterprise-5.0.64 (this should be on the Cray iso)

### Create a build root

The build is done on a normal service node, where you like (e.g. /ufs/slurm/build would work). Most scripts check for the environment variable LIBROOT. You can either edit the scripts or export this variable. Easiest way:

- export LIBROOT=/ufs/slurm/build
- mkdir -vp $LIBROOT
- cd $LIBROOT

### Install modulefile

This file is distributed as part the SLURM tar-ball in contribs/cray/opt_modulefiles_slurm. Install it as /opt/modulefiles/slurm (or anywhere else in your module path). It means that you can use Munge as soon as it is built.

### Build and install Munge

Munge is the authentication daemon and needed by SLURM.

- cp munge_build_script.sh $LIBROOT
- mkdir -vp ${LIBROOT}/munge/zip
- download munge-0.5.9.tar.bz2 or newer from http://code.google.com/p/munge/downloads/list
- copy that into ${LIBROOT}/munge/zip
- run ./munge_build_script.sh

This generates a tar-ball called $LIBROOT/munge_build-.*YYYY-MM-DD.tar.gz which has already the correct paths - in particular, the libmunge.* need to reside in /usr/lib64 since SLURM plugins call them.

Install tar-ball by

- scp $LIBROOT/munge_build-.*YYYY-MM-DD.tar.gz boot:
- ssh boot
- tar -zxvf munge_build-*.tar.gz -C /rr/current

### Configure Munge

The following steps apply to each service node where

- the slurmd or slurmctld daemon will run and/or
- users will be submitting jobs

1. on each login host create directory hierarchy:
   mkdir --mode=0711 -vp /var/lib/munge
   mkdir --mode=0700 -vp /var/log/munge
   mkdir --mode=0755 -vp /var/run/munge
2. module load slurm # as root, to get the right slurm paths
3. munged --key-file /opt/slurm/munge/etc/munge.key
4. try a "munge -n" to see if munged accepts input

When done, verify network connectivity by executing
$ munge -n |ssh other-login-host /opt/slurm/munge/bin/unmunge

If you decide to keep the installation, you may be interested in automating the process using an init.d script distributed with the Munge as src/etc/munge.init. This is installed by

- cp munge.init /rr/current/software/munge
- xtopview -c login
- cp /software/munge /etc/init.d
- chkconfig munge on

### Build libasil

This is the last step in the build preparations and sets up the (currently still external) ALPS library which does the XML-RPC calls for SLURM. This file is distributed as part the SLURM tar-ball in contribs/cray/libasil.tar.gz.

- cd $LIBROOT
- tar zxvf libasil.tar.gz
- make -C libasil

This should build the required library $LIBROOT/libasil/libasil.a. There are a number of subdirectories with test programs. If you want, you can run various test programs via

- make -C $LIBROOT/libasil/alps_tests all
- make -C $LIBROOT/libasil/sdb_tests all

There is one program which will be used in the next part. It prints the correct list of compute nodes for use in SLURM's slurm.conf file. This is the libasil/sdb_tests/tuxadmin/tuxadmin program, called with the --slurm.conf argument. Please make sure that it is working at this stage. A sample execution is shown below:
gele1>libasil/sdb_tests/tuxadmin/tuxadmin --slurm.conf
slurm can use MaxNodes=16 compute nodes. These should be listed in slurm.conf as:
NodeName=nid000[16-31]

### Build and Configure SLURM

SLURM can be built and installed as on any other computer as described Quick Start Administrator Guide. When building SLURM's slurm.conf configuration files, use the NodeName values reported by the tuxadmin command as described above. Note that the NodeAddr and NodeHostName fields should not be configured, but will be set by SLURM using data from ALPS. NodeAddr be set to the node's XYZ coordinate and be used by SLURM's smap and sview commands. NodeHostName will be set to the node's component label. The format of the component label is "c#-#c#s#n#" where the "#" fields represent in order: cabinet, row, cate, blade or slot, and node. For example "c0-1c2s5n3" is cabinet 0, row 1, cage 3, slot 5 and node 3.

The slurmd daemons will not execute on the compute nodes, but will execute on one or more front end nodes. It is from here that batch scripts will execute aprun commands to launch tasks. This is specified in the slurm.conf file by using the FrontendName and optionally the FrontEndAddr fields as seen in the examples below.

You need to specify the appropriate resource selection plugin (the SelectType option in SLURM's slurm.conf configuration file). Configure SelectType to select/cray The select/cray plugin provides an interface to ALPS plus issues calls to the select/linear, which selects resources for jobs using a best-fit algorithm to allocate whole nodes to jobs (rather than individual sockets, cores or threads).

The last Cray-specific configuration parameter controls the system topology information used to optimize resource allocations and must be configured as TopoType=topology/node_rank. The topology/node_rank plugin orders the nodes such that nodes which are close in three dimensional space a put close in one dimensional space. Since SLURM uses a best-fit algorithm in one dimensional space, the resulting job allocations should achieve good locality in three dimensional space also. For more information about how this is achieved, see SLURM's Topology Guide.

    # slurm.conf for Cray XT system of with 512 compute nodes
    # Parameters removed here
    SelectType=select/cray
    TopoType=topology/node_rank
    FrontendName=front_end[00-03]
    NodeName=nid[00000-00511] Procs=8 RealMemory=2048
    PartitionName=batch Nodes=nid[00000-00511] Default=Yes State=UP

Last modified 20 February 2011