# Heterogeneous Job Support

- Overview
- Submitting Jobs
- Managing Jobs
- Limitations

## Overviewe

Slurm version 17.11 and later supports the ability to submit and manage heterogeneous jobs, in which each component has virtually all job options available including partition, account and QOS (Quality Of Service). For example, part of a job might require four cores and 4 GB for each of 128 tasks while another part of the job would require 16 GB of memory and one CPU.

## Submitting Jobs

The salloc, sbatch and srun commands can all be used to submit heterogeneous jobs. Resource specifications for each component of the heterogeneous should be separated with ":" character. For example:

    $ sbatch --cpus-per-task=4 --mem-per-cpu=1  --ntasks=128 : \
             --cpus-per-task=1 --mem-per-cpu=16 --ntasks=1 my.bash

Environment variables used to specify default options for the job submit command will be applied to every component of the heterogeneous job.

## Managing Jobs

Information maintained in Slurm for a heterogeneous job includes:

- job_id: Each component of a heterogeneous job will have its own unique job_id.
- pack_job_id: This identification number applies to all components of the heterogeneous job. All components of the same job will have the same pack_job_id value and it will be equal to the job_id of the first component. We refer to this as the "pack leader".
- pack_job_id_set: Regular expression identifying all job_id values associated with the job.
- pack_job_offset: A unique sequence number applied to each component of the heterogeneous job. The first component will have a >pack_job_offset value of 0, the next a value of 1, etc.

job_id | pack_job_id | pack_job_offset | pack_job_id_set
--- | --- | --- | ---
123 | 123 | 0 | 123-127
124 | 123 | 1 | 123-127
125 | 123 | 2 | 123-127
126 | 123 | 3 | 123-127
127 | 123 | 4 | 123-127

Table 1: Example job IDs

The smap, smap, squeue and sview report the components of a heterogeneous job using the format "<pack_job_id>+<pack_job_offset>". For example "123+4" would represent heterogeneous job id 123 and it's fifth component (note: the first component has a pack_job_offsetvalue of 0).

A request for a specific job ID that identifes a ID of the first component of a heterogenous job (i.e. the "pack leader" will return information about all components of that job. For example:

    $ squeue --job=93
    JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST
     93+0     debug  bash  adam  R 18:18      1 nid00001
     93+1     debug  bash  adam  R 18:18      1 nid00011
     93+2     debug  bash  adam  R 18:18      1 nid00021

A request to cancel or otherwise signal a pack leader will be applied to all components of that pack job. A request to cancel a specific component of the pack job using the "#+#" notation will apply on to that specific component. For example:

    $ squeue --job=93
    JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST
     93+0     debug  bash  adam  R 19:18      1 nid00001
     93+1     debug  bash  adam  R 19:18      1 nid00011
     93+2     debug  bash  adam  R 19:18      1 nid00021
    $ scancel 93+1
    $ squeue --job=93
    JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST
     93+0     debug  bash  adam  R 19:38      1 nid00001
     93+2     debug  bash  adam  R 19:38      1 nid00021
    $ squeue --job=93
    JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST

## Limitations

In a federation of clusters, a heterogeneous job will execute entirely on the cluster from which the job is submitted. The heterogeneous job will not be eligiable to migrate between clusters or to have different components of the job execute on different clusters in the federation.

Heterogeneous jobs are only scheduled by the backfill scheduler plugin. The more frequently executed scheduling logic only starts jobs on a first-in first-out (FIFO) basis and lacks logic for concurrently scheduling all components of a heterogeneous job.

Job arrays of heterogeneous jobs are not supported.

Last modified 22 May 2017