# LEVEL_BASED Multifactor

## Contents

- Introduction
- Algorithm
- Fair share factor calculation
- Bitwise operations at each level
- Ranking
- sshare
- Configuration
- Important notes

## Introduction

PriorityFlags=LEVEL_BASED results in changes to several fair share calculations. The algorithm prioritizes users such that users in an under-served account will always have a higher fair share factor than users in an over-served account.

Some of the benefits include:

- All users from a higher priority account receive a higher fair share factor than all users from a lower priority account.
- Users are sorted and ranked to prevent errors due to precision loss.
- Account coordinators cannot accidentally harm the priority of their users relative to users in other accounts.
- Users are extremely unlikely to have exactly the same fair share factor as another user due to loss of precision in calculations.
- New jobs are immediately assigned a priority.

## Algorithm

The algorithm uses a 64-bit unsigned integer, priority_fs_raw, to store priority calculations on associations at each level. LEVEL_BASED recursively descends through the association hierarchy starting at the root. It will descend up to PriorityLevels levels below the root for calculation purposes. Any child associations that are deeper than PriorityLevels will have their fair share priority set to their parent's value.

priority_fs_raw is subdivided into "buckets" to store the fair share priority at each level. The following variables are used:

    bucket_width_in_bits = 64 / priority_levels
    unused_bucket_bits = 64 % priority_levels
    bucket_max = UINT64_MAX >> (64 - bucket_width_in_bits)

## Fair share factor calculation

As LEVEL_BASED descends through the association hierarchy, it calculates fair share in a similar way to the traditional multifactor method except that it does so at each level in the hierarchy and normalizes S beforehand.

    F = 2**(-U/S)

F
  is the association's fair share factor at the current level
U
  also known as Usage Effective, U is the association's normalized usage at the current level
S
  is the association's normalized shares at the current level

All values above are between zero and one.

See the original multifactor plugin for more information on this calculation.

## Normalization of shares before fair share calculation

One major modification to the calculation of F is that LEVEL_BASED normalizes the S value between 0.1 and 1.0 prior to the fair share calculation. This is done because a low S value can result in an F value that is approximately 0, even when U is small. An example of this behavior can be visualized by comparing an unnormalized S value between 0.0 and 1.0 to a normalized S value between 0.1 and 1.0.

Assuming 150 associations at the same level and with equal Fairshare values set in sacctmgr, S is 0.0067. When plugged into the equation, F values are often indistinguishable from each other when U is high. Compare the unnormalized S=0.0067 to a worst case S=0.1 when the value is normalized before F is calculated.

The general formula used is a linear interpolation, which maps x in range a..b to range A..B, represented by `interpolate` below.

    interpolate(x, a, b, A, B) =  (x - a) / (b - a) * (B - A) + A

## Bitwise operations at each level

After the final F calculation for the given level (assoc_level) , the value is normalized between 1 and bucket_max and stored in a temporary variable called level_fs. It is then shifted as follows:

    level_fs <<= (
                    (priority_levels - assoc_level - 1)
                    * bucket_width_in_bits
                    + unused_bucket_bits
            )

As the code recurses deeper into the tree, assoc_level is incremented from its initial value of 0. The result is a steadily decreasing shift value. level_fs is then bitwise OR'd with the parent's priority_fs_raw and stored in its own priority_fs_raw field. This fences off the effect of each level's calculation since there is no overlap.

### Example

For example, consider a four-tiered structure with colleges, departments, faculty, and users (faculty member himself/herself plus students). PriorityLevels=4 (priority_levels internally) should be set. priority_fs_raw is divided into four 16-bit buckets:

    bucket_width_in_bits = 16
    unused_bucket_bits = 0
    bucket_max = 65535

This results in the following example calculation:

        root
         ||
         \/
    Life Sciences    level_fs = 0xAAAA000000000000, priority_fs_raw = 0xAAAA000000000000
         ||
         \/
       Biology       level_fs = 0x0000123400000000, priority_fs_raw = 0xAAAA123400000000
         ||
         \/
       Dr. Bob       level_fs = 0x0000000077770000, priority_fs_raw = 0xAAAA123477770000
         ||
         \/
     Grad Student    level_fs = 0x000000000000CCCC, priority_fs_raw = 0xAAAA12347777CCCC

The final value for Grad Student is 0xAAAA12347777CCCC. This represents:

     College |  Dept   | Faculty  | User
      AAAA      1234      7777      CCCC

The priority_fs_raw values are visible as Fairshare Raw in the output of sshare -l.

## Ranking

After setting priority_fs_raw on all associations, all users are then sorted by their priority_fs_raw to obtain a ranking. This rank is then used to calculate the final fair share factor. Since this final number is stored at the association level, new jobs can be immediately assigned a priority. Note that all users, whether active or not, are included in calculations and ranked.

This step is desirable because the final fair share factor must be normalized to a floating point number between 0.0 and 1.0 that is multipled by PriorityWeightFairshare, an integer. Depending on the normalization and the value of PriorityWeightFairshare, precision loss can result in poor fair share results; users may end up with the same priorities despite widely varying usage. LEVEL_BASED solves this by sorting users then using the normalized rank as the fair share factor. Users with the same fair share factor as other users before the ranking will have the same priority as each other afterwards.

## sshare

sshare was modified to show the priority_fs_raw value as Fairshare Raw when the -l (long) parameter is specified. The field shows the hexadecimal value of priority_fs_raw for each association, thus allowing users to see the results of the fair share calculation at each level.

Note: Norm Usage is not used by LEVEL_BASED but is still displayed.

## Configuration

The following slurm.conf (SLURM_CONFIG_FILE) parameters are used to configure the LEVEL_BASED prioritization mechanism. See slurm.conf(5) man page for more details.

PriorityFlags
  Set to "LEVEL_BASED".
PriorityType
  Set this value to "priority/multifactor". The default value for this variable is "priority/basic" which enables simple FIFO scheduling.
PriorityLevels
  PriorityLevels determines how many levels of associations below the root are considered for priority purposes. Associations deeper than this number will inherit their parents' fair share priority. The number of levels must be between 1 and 16.
PriorityCalcPeriod
  PriorityCalcPeriod is the frequency in minutes that job half-life decay and LEVEL_BASED calculations are performed.

## Important Notes

- As the LEVEL_BASED algorithm ranks all users, active or not, the administrator must carefully consider how to apply other priority weights in the priority/multifactor plugin. It is worth mentioning that the PriorityWeightFairshare can be usefully set to a much smaller value than usual, possibly as low as 1 or 2 times the number of user associations.
- LEVEL_BASED requires the Slurm Accounting Database to provide the assigned shares and the consumed, computing resources described below.
- scontrol reconfigure does not cause the LEVEL_BASED algorithm to run immediately, even if switching from a different algorithm. You may have to wait until the next iteration as defined by PriorityCalcPeriod.

Last modified 20 June 2014