# Licenses Guide

## Licenses Overview

Slurm can help with software license management by assigning available licenses to jobs at scheduling time. If the licenses are not available, jobs are kept pending until licenses become available. Licenses in Slurm are essentially shared resources, meaning configured resources that are not tied to a specific host but are associated with the entire cluster.

Licenses in Slurm can be configured in two ways:

- Local Licenses: Local licenses are local to the cluster using the slurm.conf in which they are configured.
- Remote Licenses: Remote licenses are served by the database and are configured using the sacctmgr command. Remote licenses are dynamic in nature as upon running the sacctmgr command, the slurmdbd updates all clusters the licenses are assigned to.

## Local Licenses

Local licenses are defined in the slurm.conf using the Licenses option.

slurm.conf:

    Licenses=fluent:30,ansys:100

Configured licenses can be viewed using the scontrol command.

    $ scontrol show lic
    LicenseName=ansys
        Total=100 Used=0 Free=100 Remote=no
    LicenseName=fluent
        Total=30 Used=0 Free=30 Remote=no

Requesting licenses is done by using the -L, or --licenses, submission option.

    $ sbatch -L ansys:2 script.sh
    Submitted batch job 5212

    $ scontrol show lic
    LicenseName=ansys
        Total=100 Used=2 Free=98 Remote=no
    LicenseName=fluent
        Total=30 Used=0 Free=30 Remote=no

## Remote Licenses

### Use Case

A site has two license servers, one serves 100 Nastran licenses provided by FlexNet and the other serves 50 Matlab licenses from Reprise License Management. The site has two clusters named "fluid" and "pdf" dedicated to run simulation jobs using both products. The managers want to split the number of Nastran licenses equally between clusters, but assign 70% of the Matlab licenses to cluster "pdf" and the remaining 30% to cluster "fluid".

### Configuring Slurm for the use case

Here we assume that both clusters have been configured correctly in the slurmdbd using the sacctmgr command.

    $ sacctmgr show clusters format=cluster,controlhost
       Cluster     ControlHost
    ---------- ---------------
         fluid     143.11.1.3
           pdf     144.12.3.2

The licenses are added using the sacctmgr command, specifying the total count of licenses and the percentage that should be allocated to each cluster. This can be done either in one step or through a multi-step process.

One step:

    $ sacctmgr add resource name=nastran cluster=fluid,pdf \
      count=100 percentallowed=50 server=flex_host
      servertype=flexlm type=license
     Adding Resource(s)
      nastran@flex_host
       Cluster - pdf        50%
       Cluster - fluid      50%
     Settings
      Name           = nastran
      Server         = flex_host
      Description    = nastran
      ServerType     = flexlm
      Count          = 100
      Type           = License

Multi-step:

    $ sacctmgr add resource name=matlab count=50 server=rlm_host \
      servertype=rlm type=license
     Adding Resource(s)
      matlab@rlm_host
     Settings
      Name           = matlab
      Server         = rlm_host
      Description    = matlab
      ServerType     = rlm
      Count          = 50
      Type           = License

    $ sacctmgr add resource name=matlab server=rlm_host
      cluster=pdf percentallowed=70
    Adding Resource(s)
      matlab@rlm_host
       Cluster - pdf        70%
     Settings
      Server         = rlm_host
      Type           = License

    $ sacctmgr add resource name=matlab server=rlm_host
      cluster=fluid percentallowed=30
     Adding Resource(s)
      matlab@rlm_host
       Cluster - fluid     30%
     Settings
      Server         = rlm_host
      Type           = License

The sacctmgr command will now display the grand total of licenses.

    $ sacctmgr show resource
          Name     Server     Type  Count % Allocated ServerType
    ---------- ---------- -------- ------ ----------- ----------
       nastran  flex_host  License    100         100     flexlm
        matlab   rlm_host  License     50         100        rlm

    $sacctmgr show resource withclusters
          Name     Server     Type  Count % Allocated ServerType    Cluster  % Allowed
    ---------- ---------- -------- ------ ----------- ---------- ---------- ----------
       nastran  flex_host  License    100         100     flexlm        pdf         50
       nastran  flex_host  License    100         100     flexlm      fluid         50
        matlab   rlm_host  License     50         100        rlm        pdf         70
        matlab   rlm_host  License     50         100        rlm      fluid         30

The configured licenses are now visible on both clusters using the scontrol command.

    $ scontrol show lic
    LicenseName=matlab@rlm_host
        Total=35 Used=0 Free=35 Remote=yes
    LicenseName=nastran@flex_host
        Total=50 Used=0 Free=50 Remote=yes

    # On cluster "fluid":
    $ scontrol show lic
    LicenseName=matlab@rlm_host
        Total=15 Used=0 Free=15 Remote=yes
    LicenseName=nastran@flex_host
        Total=50 Used=0 Free=50 Remote=yes

When submitting jobs to remote licenses, the name and server must be used.

    $ sbatch -L nastran@flex_host script.sh
    Submitted batch job 5172

License percentages and counts can be modified as shown below:

    $ sacctmgr modify resource name=matlab server=rlm_host set \
      count=200
     Modified server resource ...
      matlab@rlm_host
      Cluster - pdf         - matlab@rlm_host
      Cluster - fluid       - matlab@rlm_host

    $ sacctmgr modify resource name=matlab server=rlm_host \
      cluster=pdf set percentallowed=60
     Modified server resource ...
      Cluster - pdf         - matlab@rlm_host

    $ sacctmgr show resource withclusters
          Name     Server     Type  Count % Allocated ServerType    Cluster  % Allowed
    ---------- ---------- -------- ------ ----------- ---------- ---------- ----------
       nastran  flex_host  License    100         100     flexlm        pdf         50
       nastran  flex_host  License    100         100     flexlm      fluid         50
        matlab   rlm_host  License    200          90        rlm        pdf         60
        matlab   rlm_host  License    200          90        rlm      fluid         30

Licenses can be deleted either on the cluster or all together as shown:

    $ sacctmgr delete resource where name=matlab server=rlm_host \
      cluster=fluid
     Deleting resource(s)...
      Cluster - fluid      - matlab@rlm_host

    $ sacctmgr delete resource where name=nastran server=flex_host
     Deleting resource(s)...
      nastran@flex_host
      Cluster - pdf        - nastran@flex_host
      Cluster - fluid      - nastran@flex_host

    $ sacctmgr show resource withclusters
          Name     Server     Type  Count % Allocated ServerType    Cluster  % Allowed
    ---------- ---------- -------- ------ ----------- ---------- ---------- ----------
        matlab   rlm_host  License    200          60        rlm      pdf         60

## Dynamic licenses

When the license counter and percentage is updated using the sacctmgr command, the values are updated in the database and the updated values are sent to the slurmctld daemon. It is possible for sites to write a script that detects global license counter changes due to new licenses being added, or old licenses being removed, and updates Slurm. Should the license count decrease, the running jobs will not be affected, only the dispatch of new jobs will reflect the new license count.

Last modified 25 June 2019