# Frequently Asked Questions

## For Users

1. Why is my job/node in COMPLETING state?
2. Why do I see the error "Can't propagate RLIMIT_..."?
3. Why is my job not running?
4. Why does the srun --overcommit option not permit multiple jobs to run on nodes?
5. Why is my job killed prematurely?
6. Why are my srun options ignored?
7. Why are "Invalid job credential" errors generated?
8. Why is the SLURM backfill scheduler not starting my job?
9. How can I run multiple jobs from within a single script?

## For Administrators

1. How is job suspend/resume useful?
2. How can I configure SLURM to use the resources actually found on a node rather than what is defined in slurm.conf?
3. Why is a node shown in state DOWN when the node has registered for service?
4. What happens when a node crashes?
5. How can I control the execution of multiple jobs per node?
6. When the SLURM daemon starts, it prints "cannot resolve X plugin operations" and exits. What does this mean?
7. Why are user tasks intermittently dying at launch with SIGPIPE error messages?
8. How can I dry up the workload for a maintenance period?

## For Users

1. Why is my job/node in COMPLETING state?
When a job is terminating, both the job and its nodes enter the state "completing." As the SLURM daemon on each node determines that all processes associated with the job have terminated, that node changes state to "idle" or some other appropriate state. When every node allocated to a job has determined that all processes associated with it have terminated, the job changes state to "completed" or some other appropriate state. Normally, this happens within a fraction of one second. However, if the job has processes that cannot be terminated with a SIGKILL, the job and one or more nodes can remain in the completing state for an extended period of time. This may be indicative of processes hung waiting for a core file to complete I/O or operating system failure. If this state persists, the system administrator should use the scontrol command to change the node's state to DOWN (e.g. "scontrol update NodeName=name State=DOWN Reason=hung_completing"), reboot the node, then reset the node's state to IDLE (e.g. "scontrol update NodeName=name State=RESUME").

2. Why do I see the error "Can't propagate RLIMIT_..."?
When the srun command executes, it captures the resource limits in effect at that time. These limits are propagated to the allocated nodes before initiating the user's job. If the soft resource limits on the job submit host are higher than the hard resource limits on the allocated host, SLURM will be unable to propagate the resource limits and print an error of the type shown above. It is recommended that the system administrator establish uniform hard resource limits on all nodes within a cluster to prevent this from occurring.

3. Why is my job not running?
The answer to this question depends upon the scheduler used by SLURM. Executing the command

> scontrol show config | grep SchedulerType

will supply this information. If the scheduler type is builtin, then jobs will be executed in the order of submission for a given partition. Even if resources are available to initiate your job immediately, it will be deferred until no previously submitted job is pending. If the scheduler type is backfill, then jobs will generally be executed in the order of submission for a given partition with one exception: later submitted jobs will be initiated early if doing so does not delay the expected execution time of an earlier submitted job. In order for backfill scheduling to be effective, users jobs should specify reasonable time limits. If jobs do not specify time limits, then all jobs will receive the same time limit (that associated with the partition), and the ability to backfill schedule jobs will be limited. The backfill scheduler does not alter job specifications of required or excluded nodes, so jobs which specify nodes will substantially reduce the effectiveness of backfill scheduling. See the backfill section for more details. If the scheduler type is wiki, this represents The Maui Scheduler. Please refer to its documentation for help. For any scheduler, you can check priorities of jobs using the command scontrol show job.

4. Why does the srun --overcommit option not permit multiple jobs to run on nodes?
The --overcommit option is a means of indicating that a job or job step is willing to execute more than one task per processor in the job's allocation. For example, consider a cluster of two processor nodes. The srun execute line may be something of this sort

> srun --ntasks=4 --nodes=1 a.out

This will result in not one, but two nodes being allocated so that each of the four tasks is given its own processor. Note that the srun --nodes option specifies a minimum node count and optionally a maximum node count. A command line of

> srun --ntasks=4 --nodes=1-1 a.out

would result in the request being rejected. If the --overcommit option is added to either command line, then only one node will be allocated for all four tasks to use.

More than one job can execute simultaneously on the same nodes through the use of srun's --shared option in conjunction with the Shared parameter in SLURM's partition configuration. See the man pages for srun and slurm.conf for more information.

5. Why is my job killed prematurely?
SLURM has a job purging mechanism to remove inactive jobs (resource allocations) before reaching its time limit, which could be infinite. This inactivity time limit is configurable by the system administrator. You can check it's value with the command

> scontrol show config | grep InactiveLimit

The value of InactiveLimit is in seconds. A zero value indicates that job purging is disabled. A job is considered inactive if it has no active job steps or if the srun command creating the job is not responding. In the case of a batch job, the srun command terminates after the job script is submitted. Therefore batch job pre- and post-processing is limited to the InactiveLimit. Contact your system administrator if you believe the InactiveLimit value should be changed.

6. Why are my srun options ignored?
Everything after the command srun is examined to determine if it is a valid option for srun. The first token that is not a valid option for srun is considered the command to execute and everything after that is treated as an option to the command. For example:

> srun -N2 hostname -pdebug

srun processes "-N2" as an option to itself. "hostname" is the command to execute and "-pdebug" is treated as an option to the hostname command. Which will change the name of the computer on which SLURM executes the command - Very bad, Don't run this command as user root!

7. Why are "Invalid job credential" errors generated?
This error is indicative of SLURM's job credential files being inconsistent across the cluster. All nodes in the cluster must have the matching public and private keys as defined by JobCredPrivateKey and JobCredPublicKey in the slurm configuration file slurm.conf.

8. Why is the SLURM backfill scheduler not starting my job?
There are significant limitations in the current backfill scheduler plugin. It was designed to perform backfill node scheduling for a homogeneous cluster. It does not manage scheduling on individual processors (or other consumable resources). It also does not update the required or excluded node list of individual jobs. These are the current limiations. You can use the scontrol show command to check if these conditions apply.

- partition: State=UP
- partition: RootOnly=NO
- partition: Shared=NO
- job: ReqNodeList=NULL
- job: ExcNodeList=NULL
- job: Contiguous=0
- job: Features=NULL
- job: MinProcs, MinMemory, and MinTmpDisk satisfied by all nodes in the partition
- job: MinProcs or MinNodes not to exceed partition's MaxNodes

As soon as any priority-ordered job in the partition's queue fail to satisfy the request, no lower priority job in that partition's queue will be considered as a backfill candidate. Any programmer wishing to augment the existing code is welcome to do so.

9. How can I run multiple jobs from within a single script?
A SLURM job is just a resource allocation. You can execute many job steps within that allocation, either in parallel or sequentially. Some jobs actually launch thousands of job steps this way. The job steps will be allocated nodes that are not already allocated to other job steps. This essential provides a second level of resource management within the job for the job steps.

## For Administrators

1. How is job suspend/resume useful?
Job suspend/resume is most useful to get particularly large jobs initiated in a timely fashion with minimal overhead. Say you want to get a full-system job initiated. Normally you would need to either cancel all running jobs or wait for them to terminate. Canceling jobs results in the loss of their work to that point from either their beginning or last checkpoint. Waiting for the jobs to terminate can take hours, depending upon your system configuration. A more attractive alternative is to suspend the running jobs, run the full-system job, then resume the suspended jobs. This can easily be accomplished by configuring a special queue for full-system jobs and using a script to control the process. The script would stop the other partitions, suspend running jobs in those partitions, and start the full-system partition. The process can be reversed when desired. One can effectively gang schedule (time-slice) multiple jobs using this mechanism, although the algorithms to do so can get quite complex. Suspending and resuming a job makes use of the SIGSTOP and SIGCONT signals respectively, so swap and disk space should be sufficient to accommodate all jobs allocated to a node, either running or suspended.

2. How can I configure SLURM to use the resources actually found on a node rather than what is defined in slurm.conf?
SLURM can either base it's scheduling decisions upon the node configuration defined in slurm.conf or what each node actually returns as available resources. This is controlled using the configuration parameter FastSchedule. Set it's value to zero in order to use the resources actually found on each node, but with a higher overhead for scheduling. A value of one is the default and results in the node configuration defined in slurm.conf being used. See "man slurm.conf" for more details.

3. Why is a node shown in state DOWN when the node has registered for service?
The configuration parameter ReturnToService in slurm.conf controls how DOWN nodes are handled. Set its value to one in order for DOWN nodes to automatically be returned to service once the slurmd daemon registers with a valid node configuration. A value of zero is the default and results in a node staying DOWN until an administrator explicity returns it to service using the command "scontrol update NodeName=whatever State=RESUME". See "man slurm.conf" and "man scontrol" for more details.

4. What happens when a node crashes?
A node is set DOWN when the slurmd daemon on it stops responding for SlurmdTimeout as defined in slurm.conf. The node can also be set DOWN when certain errors occur or the node's configuration is inconsistent with that defined in slurm.conf. Any active job on that node will be killed unless it was submitted with the srun option --no-kill. Any active job step on that node will be killed. See the slurm.conf and srun man pages for more information.

5. How can I control the execution of multiple jobs per node?
There are two mechanism to control this. If you want to allocate individual processors on a node to jobs, configure SelectType=select/cons_res. See Consumable Resources in SLURM for details about this configuration. If you want to allocate whole nodes to jobs, configure configure SelectType=select/linear. Each partition also has a configuration parameter Shared that enables more than one job to execute on each node. See man slurm.conf for more information about these configuration paramters.

6. When the SLURM daemon starts, it prints "cannot resolve X plugin operations" and exits. What does this mean?
This means that symbols expected in the plugin were not found by the daemon. This typically happens when the plugin was built or installed improperly or the configuration file is telling the plugin to use an old plugin (say from the previous version of SLURM). Restart the daemon in verbose mode for more information (e.g. "slurmctld -Dvvvvv").

7. Why are user tasks intermittently dying at launch with SIGPIPE error messages?
If you are using ldap or some other remote name service for username and groups lookup, chances are that the underlying libc library functions are triggering the SIGPIPE. You can likely work around this problem by setting CacheGroups=1 in your slurm.conf file. However, be aware that you will need to run "scontrol reconfigure " any time your groups database is updated.

8. How can I dry up the workload for a maintenance period?
There isn't a mechanism to tell SLURM that all jobs should be completed by a specific time. The best way to address this is to shorten the MaxTime associated with the partitions so as to avoid initiating jobs that will not have completed by the maintenance period.

Last modified 11 May 2006