# Heterogeneous Job Support

- Overview
- Submitting Jobs
- Managing Jobs
- Environment Variables
- Limitations
- System Administrator Information

## Overview

Slurm version 17.11 and later supports the ability to submit and manage heterogeneous jobs, in which each component has virtually all job options available including partition, account and QOS (Quality Of Service). For example, part of a job might require four cores and 4 GB for each of 128 tasks while another part of the job would require 16 GB of memory and one CPU.

## Submitting Jobs

The salloc, sbatch and srun commands can all be used to submit heterogeneous jobs. Resource specifications for each component of the heterogeneous job should be separated with ":" character. For example:

    $ sbatch --cpus-per-task=4 --mem-per-cpu=1  --ntasks=128 : \
             --cpus-per-task=1 --mem-per-cpu=16 --ntasks=1 my.bash

Options specified for one component of a heterogeneous job (or job step) will be used for subsequent components to the extent which is expected to be helpful. For example, --immediate and --job-name are propagated, while --ntasks and --mem-per-cpu are reset to default values for each component. A list of propagated options follows.

- --account
- --acctg-freq
- --begin
- --checkpoint
- --checkpoint-dir
- --cluster-constraint
- --clusters
- --comment
- --deadline
- --delay-boot
- --dependency
- --epilog (option available only in srun)
- --error
- --export
- --export-file
- --exclude
- --get-user-env
- --gid
- --hold
- --ignore-pbs
- --immediate
- --input
- --job-name
- --kill-on-bad-exit (option available only in srun)
- --label (option available only in srun)
- --max-exit-timeout (option available only in srun)
- --mcs-label
- --msg-timeout (option available only in srun)
- --no-allocate (option available only in srun)
- --no-requeue
- --nice
- --no-kill
- --open-mode (option available only in srun)
- --output
- --parsable
- --priority
- --profile
- --propagate
- --prolog (option available only in srun)
- --pty (option available only in srun)
- --qos
- --quiet
- --quit-on-interrupt (option available only in srun)
- --reboot
- --reservation
- --requeue
- --signal
- --slurmd-debug (option available only in srun)
- --task-epilog (option available only in srun)
- --task-prolog (option available only in srun)
- --time
- --test-only
- --time-min
- --uid
- --unbuffered (option available only in srun)
- --verbose
- --wait
- --wait-all-nodes
- --wckey
- --workdir

Environment variables used to specify default options for the job submit command will be applied to every component of the heterogeneous job (e.g. SBATCH_ACCOUNT).

Batch job options can be included in the submitted script for multiple heterogeneous job components. Each component should be separated by a line containing the line "#SBATCH packjob" as shown below.

    $ cat new.bash
    #!/bin/bash
    #SBATCH --cpus-per-task=4 --mem-per-cpu=16g --ntasks=1
    #SBATCH packjob
    #SBATCH --cpus-per-task=2 --mem-per-cpu=1g  --ntasks=8

    srun run.app

    $ sbatch new.bash

Is equivalent to the following:

    $ cat my.bash
    #!/bin/bash
    srun run.app

    $ sbatch --cpus-per-task=4 --mem-per-cpu=16g --ntasks=1 : \
             --cpus-per-task=2 --mem-per-cpu=1g  --ntasks=8 my.bash

The batch script will be executed in the first node in the first component of the heterogeneous job. For the above example, that will be the job component with 1 task, 4 CPUs and 64 GB of memory (16 GB for each of the 4 CPUs).

If a heterogeneous job is submitted to run in multiple clusters not part of a federation (e.g. "sbatch --cluster=alpha,beta ...") then the entire job will be sent to the cluster expected to be able to start all components at the earliest time.

## Managing Jobs

Information maintained in Slurm for a heterogeneous job includes:

- job_id: Each component of a heterogeneous job will have its own unique job_id.
- pack_job_id: This identification number applies to all components of the heterogeneous job. All components of the same job will have the same pack_job_id value and it will be equal to the job_id of the first component. We refer to this as the "pack leader".
- pack_job_id_set: Regular expression identifying all job_id values associated with the job.
- pack_job_offset: A unique sequence number applied to each component of the heterogeneous job. The first component will have a >pack_job_offset value of 0, the next a value of 1, etc.

job_id | pack_job_id | pack_job_offset | pack_job_id_set
--- | --- | --- | ---
123 | 123 | 0 | 123-127
124 | 123 | 1 | 123-127
125 | 123 | 2 | 123-127
126 | 123 | 3 | 123-127
127 | 123 | 4 | 123-127

Table 1: Example job IDs

The smap, smap, squeue and sview report the components of a heterogeneous job using the format "<pack_job_id>+<pack_job_offset>". For example "123+4" would represent heterogeneous job id 123 and it's fifth component (note: the first component has a pack_job_offset value of 0).

A request for a specific job ID that identifies a ID of the first component of a heterogeneous job (i.e. the "pack leader" will return information about all components of that job. For example:

    $ squeue --job=93
    JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST
     93+0     debug  bash  adam  R 18:18      1 nid00001
     93+1     debug  bash  adam  R 18:18      1 nid00011
     93+2     debug  bash  adam  R 18:18      1 nid00021

A request to cancel or otherwise signal a pack leader will be applied to all components of that pack job. A request to cancel a specific component of the pack job using the "#+#" notation will apply on to that specific component. For example:

    $ squeue --job=93
    JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST
     93+0     debug  bash  adam  R 19:18      1 nid00001
     93+1     debug  bash  adam  R 19:18      1 nid00011
     93+2     debug  bash  adam  R 19:18      1 nid00021
    $ scancel 93+1
    $ squeue --job=93
    JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST
     93+0     debug  bash  adam  R 19:38      1 nid00001
     93+2     debug  bash  adam  R 19:38      1 nid00021
    $ squeue --job=93
    JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST

Email notification for job state changes (the --mail-type option) is only supported for a pack leader. Requests for email notifications for other components of a heterogeneous job will be silently ignored.

Requests to perform the following operations a job can only be requested for a pack leader and will be applied to all components of that heterogeneous job. Requests to operate on individual components of the heterogeneous will return an error.

- requeue
- resume
- suspend

## Environment Variables

Slurm environment variables will be set independently for each component of the job by appending "_PACK_GROUP_" and a sequence number the the usual name. In addition, the "SLURM_JOB_ID" environment variable will contain the job ID of the pack leader and "SLURM_PACK_SIZE" will contain the number of components in the job. For example:

    $  salloc -N1 : -N2 bash
    salloc: Pending job allocation 11741
    salloc: job 11741 queued and waiting for resources
    salloc: job 11741 has been allocated resources
    $ env | grep SLURM
    SLURM_JOB_ID=11741
    SLURM_PACK_SIZE=2
    SLURM_JOB_ID_PACK_GROUP_0=11741
    SLURM_JOB_ID_PACK_GROUP_1=11742
    SLURM_NNODES_PACK_GROUP_0=1
    SLURM_NNODES_PACK_GROUP_1=2
    SLURM_JOB_NODELIST_PACK_GROUP_0=nid00001
    SLURM_JOB_NODELIST_PACK_GROUP_1=nid[00011-00012]
    ...

## Limitations

In a federation of clusters, a heterogeneous job will execute entirely on the cluster from which the job is submitted. The heterogeneous job will not be eligible to migrate between clusters or to have different components of the job execute on different clusters in the federation.

Job arrays of heterogeneous jobs are not supported.

The srun command's --no-allocate option is not supported for heterogeneous jobs.

Heterogeneous jobs are only scheduled by the backfill scheduler plugin. The more frequently executed scheduling logic only starts jobs on a first-in first-out (FIFO) basis and lacks logic for concurrently scheduling all components of a heterogeneous job.

Heterogeneous jobs are not supported with Slurm's select/serial plugin.

Heterogeneous jobs are not supported Cray ALPS systems.

## System Administrator Information

The job submit plugin is invoked independently for each component of a heterogeneous job.

Last modified 19 June 2017