# Heterogeneous Job Support

- Overview
- Submitting Jobs
- Managing Jobs
- Limitations

## Overviewe

Slurm version 17.11 and later supports the ability to submit and manage heterogeneous jobs, in which each component has virtually all job options available including partition, account and QOS (Quality Of Service). For example, part of a job might require four cores and 4 GB for each of 128 tasks while another part of the job would require 16 GB of memory and one CPU.

## Submitting Jobs

The salloc, sbatch and srun commands can all be used to submit heterogeneous jobs. Resource specifications for each component of the heterogeneous should be separated with ":" character. For example:

    $ sbatch --cpus-per-task=4 --mem-per-cpu=1  --ntasks=128 : \
             --cpus-per-task=1 --mem-per-cpu=16 --ntasks=1 my.bash

Environment variables used to specify default options for the job submit command will be applied to every component of the heterogeneous job.

## Managing Jobs

Information maintained in Slurm for a heterogeneous job includes:

- job_id: Each component of a heterogeneous job will have its own unique job_id.
- pack_job_id: This identification number applies to all components of the heterogeneous job. All components of the same job will have the same pack_job_id value and it will be equal to the job_id of the first component.
- pack_job_id_set: Regular expression identifying all job_id values associated with the job.
- pack_job_offset: A unique sequence number applied to each component of the heterogeneous job. The first component will have a >pack_job_offset value of 0, the next a value of 1, etc.

job_id | pack_job_id | pack_job_offset | pack_job_id_set
--- | --- | --- | ---
123 | 123 | 0 | 123-127
124 | 123 | 1 | 123-127
125 | 123 | 2 | 123-127
126 | 123 | 3 | 123-127
127 | 123 | 4 | 123-127

Table 1: Example job IDs

The smap, smap, squeue and sview report the components of a heterogeneous job using the format "<pack_job_id>+<pack_job_offset>". For example "123+4" would represent heterogeneous job id 123 and it's fifth component (note: the first component has a pack_job_offsetvalue of 0).

A request for a specific job ID that identifes a ID of the first component of a heterogenous job will return information about all pack job components. For example:

    $ squeue --job=93
    JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST
     93+0     debug  bash  adam  R 18:18      1 nid00001
     93+1     debug  bash  adam  R 18:18      1 nid00011
     93+2     debug  bash  adam  R 18:18      1 nid00021

## Limitations

In a federation of clusters, a heterogeneous job will execute entirely on the cluster from which the job is submitted. The heterogeneous job will not be eligiable to migrate between clusters or to have different components of the job execute on different clusters in the federation.

Heterogeneous jobs are only scheduled by the backfill scheduler plugin. The more frequently executed scheduling logic only starts jobs on a first-in first-out (FIFO) basis and lacks logic for concurrently scheduling all components of a heterogeneous job.

Job arrays of heterogeneous jobs are not supported.

Last modified 22 May 2017