# SLURM Troubleshooting Guide

This guide is meant as a tool to help system administrators or operators troubleshoot SLURM failures and restore services.

- SLURM is not responding
- Jobs are not getting scheduled
- Notes are getting set to a DOWN state
- Networking problems

## SLURM is not responding

1. Execute "scontrol ping" to determine if the primary and backup controllers are responding.
2. If it responds for you, this could be a networking or configuration problem specific to some user or node in the cluster.
3. If not responding, directly login to the machine and try again to rule out network and configuration problems.
4. If still not responding, check if there is an active slurmctld dameon by executing "ps -el | grep slurmctld".
5. If slurmctld is not running, restart it (typically as user root using the command "/etc/init.d/slurm start"). You should check the log file (SlurmctldLog in the slurm.conf file) for an indication of why it failed. If it keeps failing, you should contact the slurm team for help at slurm-dev@lists.llnl.gov.
6. If slurmctld is running but not responding (a very rare situation), then kill and restart it (typically as user root using the commands "/etc/init.d/slurm stop" and then "/etc/init.d/slurm start").
7. If it hangs again, increase the verbosity of debug messages (increase SlurmctldDebug in the slurm.conf file) and restart. Again check the log file for an indication of why it failed. At this point, you should contact the slurm team for help at slurm-dev@lists.llnl.gov.
8. If it continues to fail without an indication as to the failure mode, restart without preserving state (typically as user root using the commands "/etc/init.d/slurm stop" and then "/etc/init.d/slurm startclean"). Note: All running jobs and other state information will be lost.

## Jobs are not getting scheduled

## Notes are getting set to a DOWN state

## Networking and configuration problems

1. Check the controller and/or slurmd log files (SlurmctldLog and SlurmdLog in the slurm.conf file) for an indication of why it is failing.
2. Check for consistent slurm.conf and credential files on the node(s) experiencing problems.
3. If this is user-specific problem, check that the user is configured on the controller computer(s) as well as the compute nodes. The user doesn't need to be able to login, but his user ID must exist.
4. Check that a consistent version of SLURM exists on all of the nodes (execute "sinfo -V" or "rpm -qa | grep slurm"). If the first two digits of the version number match it should work fine, but version 1.1 commands will not work with version 1.2 daemons or vise-versa.

Last modified 12 October 2006