# Plane distribution: -m plane=x

The plane distribution results in a block cyclic distribution with block size x. We chose to call the block cyclic distribution a plane distribution due to how we place the distribution onto logical processors in the multi-core environment. See Section "Plane distribution and task affinity".

To specify the plane distribution add to the srun command line --distribution=plane=x where x is the requested plane/block size.

More information on block cyclic distributions is available at: http://www.netlib.org/scalapack/slug/node75.html.

## Examples of plane (block cyclic) distributions

In the examples below we assume we have 21 tasks and that the task list is: 0, 1, 2, 3, 4, ..., 19, 20.

On One (1) node: srun -N 1-1 -n 21 -m plane=4 -s <...>.

The distribution results in a block distribution of blocksize 21. Even thought the user specified a plane size of 4 the final plane distribution results in a block distribution of blocksize 21.

Figure 1: Process layout for srun -N 1-1 -n 21 -m plane=4 -s <...>

On four (4) nodes: srun -N 4-4 -n 21 -m plane=4 -s <...>.

The plane distribution with a plane size of 4 results in the following allocation of the task ids:

Figure 2: Process layout for srun -N 4-4 -n 21 -m plane=4 -s <...>

On four (4) nodes: srun -N 4-4 -n 21 -m plane=2 -s <...> .

The plane distribution with a block size of 2 results in the following allocation of the task ids:

Figure 3: Process layout for srun -N 4-4 -n 21 -m plane=2 -s <...>

More information on block cyclic distributions is available at: http://www.netlib.org/scalapack/slug/node75.html.

## Assumptions and Limitations

SLURM assumes that the number of tasks divided by the plane size is equal to the number of nodes specified.

In other words if the following is true SLURM will generate an error message:

    if ((n/plane_size < min_nodes) ((N-1)*plane_size >= n))
                generate an error message: Too few processes for the
    	           requested {plane, node} distribution.

## Plane distribution and task affinity

The concept behind this distribution is to divide the clusters into planes. Each plane includes a number of the lowest level of logical processors (CPU, cores, threads depending on the architecture) on each node. We then schedule within each plane first and then across planes.

We ensure that the processes are located correctly by setting the process affinity to the specified/correct logical processor. Process affinity is available to SLURM when the task affinity plug-in is enabled.

On a dual-processor node with quad-core processors (see figure 4) the plane distribution results in:

- One plane if the plane size x=8. In this case the processors are scheduled by first filling up the nodes and then scheduled across the nodes.
- Eight planes if the plane size x=1. In this case we would always schedule across the node first.

Figure 4: Quad-core dual-processor system

In a multi-core/hyper-threaded environment, two planes would provide better locality but potentially more contention for other resources.

On the other hand, four planes (scheduling acorss processors) would minimize contention for cache and memory.

### Examples of plane distributions with process affinity enabled

In the examples below we assume we have 21 tasks and that the task list is: 0, 1, 2, 3, 4, ..., 19, 20.

On One (1) node: srun -N 1-1 -n 21 -m plane=4 -s <...>. Even thought the user specified a block size of 4 the final plane distribution results in a block distribution of blocksize 21.

Figure 5: Process layout for srun -N 1-1 -n 21 -m plane=4 -s <...>.

On four (4) nodes: srun -N 4-4 -n 21 -m plane=4 -s <...> . The plane distribution with a block size of 4 results in the following allocation of the task ids:

Figure 6: Process layout for srun -N 4-4 -n 21 -m plane=4 -s <...>.

On four (4) nodes: srun -N 4-4 -n 21 -m plane=2 -s <...> . The plane distribution with a block size of 2 results in the following allocation of the task ids:

Figure 7: Process layout for srun -N 4-4 -n 21 -m plane=2 -s <...>.

The reason Process id 9 is assigned to "Node 0: Socket 1: Core 0" and not "Node 0: Socket 0: Core 1" is that SLURM assume a cyclic across the lowest level of logical processors by default.

Last modified 9 October 2006