slaunch(1)

NAME

slaunch - Launch a parallel application under a SLURM job allocation.

SYNOPSIS

slaunch [options] <command> [command args]

DESCRIPTION

slaunch launches a parallel application (a job step in SLURM parlance) on the nodes, or subset of nodes, in a job allocation. A valid job allocation is a prerequisite of running slaunch. The ID of the job allocation may be passed to slaunch through either the --jobid command line parameter or the SLAUNCH_JOBID environment variable. The salloc and sbatch commands may be used to request a job allocation, and each of those commands automatically set the SLURM_JOB_ID environment variable, which is also understood by slaunch. Users should not set SLURM_JOB_ID on their own; use SLAUNCH_JOBID instead.

OPTIONS

--jobid <JOBID>
    The job allocation under which the parallel application should be launched. If slaunch is running under salloc or a batch script, slaunch can automatically determint the jobid from the SLURM_JOB_ID environment variable. Otherwise, you will need to tell slaunch which job allocation to use.

-n, --tasks[=]<number>
    Specify the number of processes to launch. The default is one process per node.

-N, --nodes[=]<number>
    Specify the number of nodes to be used by this job step. By default, slaunch will use all of the nodes in the specified job allocation.

-r, --relative[=]<number>
    Specify the first node in the allocation on which this job step will be launched. Counting starts at zero, thus the first node in the job allocation is node 0. The option to --relative may also be a negative number. -1 is the last node in the allocation, -2 is the next to last node, etc. By default, the controller will select the starting node (assuming that there are no other nodelist or task layout options that specify specific nodes).

-c, --cpus-per-task[=]<ncpus>
    Specify that each task requires ncpus number of CPUs. Useful for applications in which each task will launch multiple threads and can therefore benefit from there being free processors on the node.

-J, --name[=]<name>
    Set the name of the job step. By default, the job step's name will be the name of the executable which slaunch is launching.

-K, --kill-on-bad-exit
    Terminate the job step if any task exits with a non-zero exit code. By default slaunch will not terminate a job step because of a task with a non-zero exit code.

-l, --label
    Prepend each line of task standard output or standard error with the task number of its origin.

-u, --unbuffered
    Do not line buffer standard output or standard error from remote tasks. This option cannot be used with --label.

-C, --overcommit
    Permit the allocation of more tasks to a node than there are available processors. Normally SLURM will only allow up to N tasks on a node with N processors, but this option will allow more than N tasks to be assigned to a node.

-q, --quiet
    Suppress informational messages from slaunch. Errors will still be displayed.

-v, --verbose
    Increase the verbosity of slaunch's informational messages. Multiple -v's will further increase slaunch's verbosity.

-d, --slurmd-debug[=]<level>
    Specify a debug level for slurmd(8). level may be an integer value between 0 [quiet, only errors are displayed] and 4 [verbose operation]. The slurmd debug information is copied onto the stderr of the job. By default only errors are displayed.

-W, --wait[=]<seconds>
    slaunch will wait the specified number of seconds after the first tasks exits before killing all tasks in the job step. If the value is 0, slaunch will wait indefinitely for all tasks to exit. The default value is give by the WaitTime parameter in the slurm configuration file (see slurm.conf(5)). The --wait option can be used to insure that a job step terminates in a timely fashion in the event that one or more tasks terminate prematurely.

-m, --distribution=
    (block|cyclic|hostfile|plane=<options>) Specify an alternate distribution method for remote processes.

    block
        The block method of distribution will allocate processes in-order to the cpus on a node. If the number of processes exceeds the number of cpus on all of the nodes in the allocation then all nodes will be utilized. For example, consider an allocation of three nodes each with two cpus. A four-process block distribution request will distribute those processes to the nodes with processes one and two on the first node, process three on the second node, and process four on the third node. Block distribution is the default behavior if the number of tasks exceeds the number of nodes requested.

    cyclic
        The cyclic method distributes processes in a round-robin fashion across the allocated nodes. That is, process one will be allocated to the first node, process two to the second, and so on. This is the default behavior if the number of tasks is no larger than the number of nodes requested.

    plane
        The tasks are distributed in blocks of a specified size. The options include a number representing the size of the task block. This is followed by an optional specification of the task distribution scheme within a block of tasks and between the blocks of tasks. For more details (including examples and diagrams), please see http://www.llnl.gov/linux/slurm/mc_support.html and http://www.llnl.gov/linux/slurm/dist_plane.html.

    hostfile
        The hostfile method of distribution will allocate processes in-order as listed in file designated by the environment variable SLURM_HOSTFILE. If this variable is listed it will over ride any other method specified.

-w, --nodelist-byname[=]<node name list>
    Request a specific list of node names. The list may be specified as a comma-separated list of node names, or a range of node names (e.g. mynode[1-5,7,...]). Duplicate node names are not permitted in the list. The order of the node names in the list is not important; the node names will be sorted my SLURM.

-L, --nodelist-byid[=]<node index list>
    Request a specific set of nodes in a job alloction on which to run the tasks of the job step. The list may be specified as a comma-separated list relative node indices in the job allocation (e.g., "0,2-5,-2,8"). Duplicate indices are permitted, but are ignored. The order of the node indices in the list is not important; the node indices will be sorted my SLURM.

-T, --task-layout-byid[=]<node index list>
    Request a specific task layout using node indices within the job allocation. The node index list can contain duplicate indices, and the indices may appear in any order. The order of indices in the nodelist IS significant. Each node index in the list represents one task, with the Nth node index in the list designating on which node the Nth task should be launched. For example, given an allocation of nodes "linux[0-15]" and a node index list "4,-1,1-3" task 0 will run on "linux4", task 1 will run on "linux15", task 2 on "linux1", task 3 on "linux2", and task 4 on "linux3". NOTE: This option implicitly sets the task distribution method to "arbitrary". Some network switch layers do not permit arbitrary task layout.

-Y, --task-layout-byname[=]<node name list>
    Request a specific task layout. The nodelist can contain duplicate node names, and node names may appear in any order. The order of node names in the nodelist IS significant. Each node name in the nodes list represents one task, with the Nth node name in the nodelist designating on which node the Nth task should be launched. For example, a nodelist of mynode[4,3,1-2,4] means that tasks 0 and 4 will run on mynode4, task 1 will run on mynode3, task 2 will run on mynode1, and task 3 will run on mynode2. NOTE: This option implicitly sets the task distribution method to "arbitrary". Some network switch layers do not permit arbitrary task layout.

-F, --task-layout-file[=]<filename>
    Request a specific task layout. This option is much like the --task-layout-byname option, except that instead of a nodelist you supply the name of a file. The file contains a nodelist that may span multiple lines of the file. NOTE: This option implicitly sets the task distribution method to "arbitrary". Some network switch layers do not permit arbitrary task layout.

-i, --slaunch-input[=]<filename pattern>
-o, --slaunch-output[=]<filename pattern>
-e, --slaunch-error[=]<filename pattern>
    Change slaunch's standard input, standard output, or standard error to be a file of name "filename pattern". These options are similar to using shell IO redirection capabilities, but with the additional ability to replace certain symbols in the filename with useful SLURM information. Symbols are listed below. By default, slaunch broadcasts its standard input over the network to the standard input of all tasks. Likewise, standard output and standard error from all tasks are collected over the network by slaunch and printed on its standard output or standard error, respectively. If you want to see traffic from fewer tasks, see the --slaunch-[input|output|error]-filter options. Supported replacement symbols are:
    %J  Job allocation number and job step number in the form "jobid.stepid". For instance, "128.0".
    %j  Job allocation number.
    %s  Job step number.
--slaunch-input-filter[=]<task number>
--slaunch-output-filter[=]<task number>
--slaunch-error-filter[=]<task number>
    Only transmit standard input to a single task, or print the standard output or standard error from a single task. These options perform the filtering locally in slaunch. All tasks are still capable of sending or receiving standard IO over the network, so the "sattach" command can still access the standard IO streams of the other tasks. (NOTE: for -output and -error, the streams from all tasks WILL be transmitted to slaunch, but it will only print the streams for the selected task. If your tasks print a great deal of data to standard output or error, this can be performance limiting.)

-I, --task-input[=]<filename pattern>
-O, --task-output[=]<filename pattern>
-E, --task-error[=]<filename pattern>
    Instruct SLURM to connect each task's standard input, standard output, or standard error directly to the file name specified in the "filename pattern". By default, the standard IO streams of all tasks are received and transmitted over the network to commands like slaunch and sattach. These options disable the networked standard IO streams and instead connect the standard IO streams of the tasks directly to files on the local node of each task (although the file may, of course, be located on a networked filesystem). Whether or not the tasks share a file depends on whether or not the file lives on a local filesystem or a shared network filesytem, and on whether or not the filename pattern expands to the same file name for each task. The filename pattern may contain one or more replacement symbols, which are a percent sign "%" followed by a letter (e.g. %t). Supported replacement symbols are:
    %J  Job allocation number and job step number in the form "jobid.stepid". For instance, "128.0".
    %j  Job allocation number.
    %s  Job step number.
    %N  Node name. (Will result in a separate file per node.)
    %n  Relative node index number within the job step. All nodes used by the job step will be number sequentially starting at zero. (Will result in a separate file per node.)
    %t  Task rank number. (Will result in a separate file per task.)
-D, --workdir[=]<directory>
    Set the working directory of the tasks to directory before execution. The default task working directory is slaunch's working directory.
--mpi[=]<mpi_type>
    Identify the type of MPI to be used.
--uid[=]<user>
    Attempt to submit and/or run a job as user instead of the invoking user id. The invoking user's credentials will be used to check access permissions for the target partition. User root may use this option to run jobs as a normal user in a RootOnly partition for example. If run as root, slaunch will drop its permissions to the uid specified after node allocation is successful. user may be the user name or numerical user ID.
--gid[=]<group>
    If slaunch is run as root, and the --gid option is used, submit the job with group's group access permissions. group may be the group name or the numerical group ID.
--core[=]<type>
    Adjust corefile format for parallel job. If possible, slaunch will set up the environment for the job such that a corefile format other than full core dumps is enabled. If run with type = "list", slaunch will print a list of supported corefile format types to stdout and exit.
--propagate[=rlimits]
    Allows users to specify which of the modifiable (soft) resource limits to propagate to the compute nodes and apply to their jobs. If rlimits is not specified, then all resource limits will be propagated.
--prolog[=]<executable>
    slaunch will run executable just before launching the job step. The command line arguments for executable will be the command and arguments of the job step. If executable is "none", then no prolog will be run. This parameter overrides the SrunProlog parameter in slurm.conf.
--epilog[=]<executable>
    slaunch will run executable just after the job step completes. The command line arguments for executable will be the command and arguments of the job step. If executable is "none", then no epilog will be run. This parameter overrides the SrunEpilog parameter in slurm.conf.
--task-prolog[=]<executable>
    The slurmd daemon will run executable just before launching each task. This will be executed after any TaskProlog parameter in slurm.conf is executed. Besides the normal environment variables, this has SLURM_TASK_PID available to identify the process ID of the task being started. Standard output from this program of the form "export NAME=value" will be used to set environment variables for the task being spawned.
--task-epilog[=]<executable>
    The slurmd daemon will run executable just after each task terminates. This will be before after any TaskEpilog parameter in slurm.conf is executed. This is meant to be a very short-lived program. If it fails to terminate within a few seconds, it will be killed along with any descendant processes.
--comm-hostname[=]<hostname|address>
    Specify the hostname or address to be used for PMI communications only (MPCIH2 communication bootstrapping mechanism). Defaults to short hostname of the node on which slaunch is running.
--network[=]<options>
    (NOTE: this option is currently only of use on AIX systems.) Specify the communication protocol to be used. The interpretation of type is system dependent. For AIX systems with an IBM Federation switch, the following comma-separated and case insensitive options are recongnized: IP (the default is user-space), SN_ALL, SN_SINGLE, BULK_XFER and adapter names. For more information, on IBM systems see poe documenation on the environment variables MP_EUIDEVICE and MP_USE_BULK_XFER.
--multi-prog
    This option allows one to launch tasks with different executables within the same job step. When this option is present, slaunch no long accepts the name of an executable "command" on the command line, instead it accepts the name of a file. This file specifies which executables and command line parameters should be used by each task in the job step. See the section MULTIPLE PROGRAMS FILE below for an explanation of the multiple program file syntax.
-h, --help
    Display help information and exit.
-u, --usage
    Display brief usage message and exit.
-V, --version
    Display version information and exit.
Affinity/Multi-core Options (when the task/affinity or task/numa plugin is enabled):
--cpu_bind=[{quiet,verbose},]type
    Bind tasks to CPUs
    q[uiet],
        quietly bind before task runs (default)
    v[erbose],
        verbosely report binding before task runs
    no[ne]
        don't bind tasks to CPUs (default)
    rank
        bind by task rank
    map_cpu:<list>
        bind by mapping CPU IDs to tasks as specified where <list> is <cpuid1>,<cpuid2>,...<cpuidN>. CPU IDs are interpreted as decimal values unless they are preceded with '0x' in which case they interpreted as hexadecimal values.
    mask_cpu:<list>
        bind by setting CPU masks on tasks as specified where <list> is <mask1>,<mask2>,...<maskN>. CPU masks are always interpreted as hexadecimal values but can be preceded with an optional '0x'.
To have SLURM always report on the selected CPU binding for all srun commands executed in a shell, you can also enable verbose mode separately from the command line with:
        setenv SLURM_CPU_BIND verbose
SLURM_CPU_BIND will not propagate into the tasks environment (binding by default only affects the first srun). To propagate --cpu_bind to successive srun commands, first do the following in each task:
 setenv SLURM_CPU_BIND \
   ${SLURM_CPU_BIND_VERBOSE},${SLURM_CPU_BIND_TYPE}${SLURM_CPU_BIND_LIST}
Affinity/Multi-core Options (when the task/affinity plugin is enabled and the NUMA memory functions are available):
--mem_bind=[{quiet,verbose},]type
    Bind tasks to memory. Note that the resolution of CPU and memory binding may differ on some architectures. For example, CPU binding may be performed at the level of the cores within a processor while memory binding will be performed at the level of nodes, where the definition of "nodes" may differ from system to system. The use of any type other than "none" or "local" is not recommended. If you want greater control, try running a simple test code with the options "--cpu_bind=verbose,none --mem_bind=verbose,none" to determine the specific configuration.
    q[uiet],
        quietly bind before task runs (default)
    v[erbose],
        verbosely report binding before task runs
    no[ne]
        don't bind tasks to memory (default)
    rank
        bind by task rank (not recommended)
    local
        Use memory local to the processor in use
    map_mem:<list>
        bind by mapping a node's memory to tasks as specified where <list> is <cpuid1>,<cpuid2>,...<cpuidN>. CPU IDs are interpreted as decimal values unless they are preceded with '0x' in which case they interpreted as hexadecimal values (not recommended)
    mask_mem:<list>
        bind by setting memory masks on tasks as specified where <list> is <mask1>,<mask2>,...<maskN>. memory masks are always interpreted as hexadecimal values but can be preceded with an optional '0x' (not recommended)

To have SLURM always report on the selected memory binding for all srun commands executed in a shell, you can also enable verbose mode separately from the command line with:
        setenv SLURM_MEM_BIND verbose
SLURM_MEM_BIND will not propagate into the tasks environment (binding by default only affects the first srun). To propagate --mem_bind to successive srun commands, first do the following in each task:
 setenv SLURM_MEM_BIND \
   ${SLURM_MEM_BIND_VERBOSE},${SLURM_MEM_BIND_TYPE}${SLURM_MEM_BIND_LIST}

INPUT ENVIRONMENT VARIABLES
Some slaunch options may be set via environment variables. These environment variables, along with their corresponding options, are listed below. Note: Command line options will always override environment variables settings.
SLAUNCH_COMM_HOSTNAME
    Same as --comm-hostname.
SLAUNCH_CORE_FORMAT
    Same as --core.
SLAUNCH_CPU_BIND
    Same as --cpu_bind.
SLAUNCH_DEBUG
    Same as -v or --verbose.
SLAUNCH_DISTRIBUTION
    Same as -m or --distribution.
SLAUNCH_JOBID
    Same as --jobid.
SLAUNCH_KILL_BAD_EXIT
    Same as -K or --kill-on-bad-exit.
SLAUNCH_LABELIO
    Same as -l or --label.
SLAUNCH_MEM_BIND
    Same as --mem_bind.
SLAUNCH_MPI_TYPE
    Same as --mpi.
SLAUNCH_OVERCOMMIT
    Same as -C or --overcomit.
SLAUNCH_WAIT
    Same as -W or --wait.
SLURMD_DEBUG
    Same as -d or --slurmd-debug

OUTPUT ENVIRONMENT VARIABLES
slaunch will set the following environment variables which will appear in the environments of all tasks in the job step. Since slaunch sets these variables itself, they will also be available to --prolog and --epilog scripts. (Notice that the "backwards compatibility" environment variables clobber some of the variables that were set by salloc or sbatch at job allocation time. The newer SLURM_JOB_* and SLURM_STEP_* names do not conflict, so any task in any job step can easily determine the parameters of the job allocation.)
SLURM_STEP_ID (and SLURM_STEPID for backwards compatibility)
    The ID of the job step within the job allocation.
SLURM_STEP_NODELIST
    The list of nodes in the job step.
SLURM_STEP_NUM_NODES (and SLURM_NNODES for backwards compatibility)
    The number of nodes used by the job step.
SLURM_STEP_NUM_TASKS (and SLURM_NPROCS for backwards compatibility)
    The number of tasks in the job step.
SLURM_STEP_TASKS_PER_NODE (and SLURM_TASKS_PER_NODE for backwards compatibility)
    The number of tasks on each node in the job step.
SLURM_STEP_LAUNCHER_HOSTNAME (and SLURM_SRUN_COMM_HOST for backwards compatibility)
SLURM_STEP_LAUNCHER_PORT (and SLURM_SRUN_COMM_PORT for backwards compatibility)
Additionally, SLURM daemons will ensure that the the following variables are set in the environments of all tasks in the job step. Many of the following variables will have different values in each task's environment. (These variables are not available to the slaunch --prolog and --epilog scripts.)
SLURM_NODEID
    Node ID relative to other nodes in the job step. Counting begins at zero.
SLURM_PROCID
    Task ID relative to the other tasks in the job step. Counting begins at zero.
SLURM_LOCALID
    Task ID relative to the other tasks on the same node which belong to the same job step. Counting begins at zero.
SLURMD_NODENAME
    The SLURM NodeName for the node on which the task is running. Depending on how your system administrator has configured SLURM, the NodeName for a node may not be the same as the node's hostname. When you use commands such as sinfo and squeue, or look at environment variables such as SLURM_JOB_NODELIST and SLURM_STEP_NODELIST, you are seeing SLURM NodeNames.

MULTIPLE PROGRAMS FILE
Comments in the configuration file must have a "#" in collumn one. The configuration file contains the following fields separated by white space:
Task rank
    One or more task ranks to use this configuration. Multiple values may be comma separated. Ranges may be indicated with two numbers separated with a '-'. To indicate all tasks, specify a rank of '*' (in which case you probably should not be using this option).
Executable
    The name of the program to execute. May be fully qualified pathname if desired.
Arguments
    Program arguments. The expression "%t" will be replaced with the task's number. The expression "%o" will be replaced with the task's offset within this range (e.g. a configured task rank value of "1-5" would have offset values of "0-4"). Single quotes may be used to avoid having the enclosed values interpretted. This field is optional.
For example:
###################################################################
# srun multiple program configuration file
#
# srun -n8 -l --multi-prog silly.conf
###################################################################
4-6       hostname
1,7       echo  task:%t
0,2-3     echo  offset:%o
$ srun -n8 -l --multi-prog silly.conf
0: offset:0
1: task:1
2: offset:1
3: offset:2
4: linux15.llnl.gov
5: linux16.llnl.gov
6: linux17.llnl.gov
7: task:7

EXAMPLES
To launch a job step (parallel program) in an existing job allocation:
    slaunch --jobid 66777 -N2 -n8 myprogram
To grab an allocation of nodes and launch a parallel application on one command line (See the salloc man page for more examples):
    salloc -N5 slaunch -n10 myprogram

COPYING
Copyright (C) 2006 The Regents of the University of California. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). UCRL-CODE-226842.
This file is part of SLURM, a resource management program. For details, see <http://www.llnl.gov/linux/slurm/>.
SLURM is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
SLURM is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

SEE ALSO
sinfo(1), sattach(1), salloc(1), sbatch(1), squeue(1), scancel(1), scontrol(1), slurm.conf(5), sched_setaffinity(2), numa(3)