SLURM Machine Status Infrastructure
November 26, 2001
By Moe Jette


Abstract

The purpose of SLURM's machine status infrastructure is to configure 
and monitor the state of nodes in the cluster. 
This document describes the phase one Machine Status implementation, 
which includes only basic node status information. 
More machine state information and job state information will be 
incorporated in phases two and three.
The Machine Status Manager (MSM) will execute on the SLURM Control 
Machine and record the configuration of each node along with its 
latest reported state. 
This information will be available for viewing and/or modification 
using APIs and a Machine Status Tool (MST).
The Machine Status Daemon (MSD) will execute on each SLURM node 
and report the state to the MSM.


Machine Status Manager

The Machine Status Manager (MSM) is responsible for maintaining a 
configuration record for each node in the cluster. 
MSM will have a configuration file identifying a variety of parameters. 
The location of this file will be provided by the symbolic link at 
"/etc/SLURM.cfg".
There will be default values for most parameters if not specified. 
Lines in the configuration file having "#" in column one will be 
considered comments.
Parameters used by the machine status include:
ControlMachine    The name of the machine where control functions operate
CollectorNodes    Comma separated list of nodes which can server to 
                  collect messages and combine the data so as to 
                  reduce network traffic (Note: This design feature is unique  
                  to SLURM and DPCS, it offers vastly improved scalability, 
                  default is none, specify comma separated list of node 
                  names as desired)
NodeSpecConf      Fully qualified pathname of the file containing node 
                  configuration information as described below (default 
                  "/usr/local/SLURM/NodeSpecConf")
PartitionConf     Fully qualified pathname of the file containing partitoin 
                  configuration information as described below (default 
                  "/usr/local/SLURM/PartitionConf")
DefaultPartition  Name of the default partition
MachStatusManager The qualified pathname of the file containing the Machine 
                  Status Manager (default "/usr/local/SLURM/MachStatusManager")
MachStatusDaemon  The qualified pathname of the file containing the Machine 
                  Status Daemon (default "/usr/local/SLURM/MachStatusDaemon")
MachStatusPort    The port to be used for the Machine Status Manager and 
                  Machine Status Daemon communications. Should be privileged 
                  port (acquired only by user root).
MachStatusDebug   A list of debug flags separated by commas (default is 
                  minimal logging, example "init,msg")
HeartBeatInterval Seconds between node status reports (default is "300")
HeartBeatTimeout  If last node status report is at least this number 
                  of seconds ago, node is considered "Down" (default is "600")
Only one parameter should be specified per line with the parameter's name, 
an equal sign, and the value. 
White space is ignored.
A sample SlurmConf file is included at the end of this document.


MSM will maintain the following information about each node:
 1 Name           Name of a node as returned by hostname (e.g. "lx12")
 2 OS             Operating System name and level (output of the command
                  "/bin/uname -s -r | /bin/sed 's/ /./g'")
 3 CPUs           Number of processors (e.g. "2")
 4 Speed          Relative speed of these CPUs, units can be an arbitrary 
                  floating point number, but MHz value is recommended 
                  (e.g. "863.8")
 5 RealMemory     Size of real memory in MegaBytes (e.g. "2048")
 6 VirtualMemory  Size of virtual memory in MegaBytes (e.g. "4096")
 7 TmpDisk        Size of temporary disk storage in MegaBytes (e.g. "16384")
 8 Partition      List of partition numbers (collections of nodes) this node 
                  belongs to, partition numbers range from 0 to 31 and are 
                  specified with comma separators (e.g. "1,3").
 9 LastResponse   Time of last contact from node, format is time_t as 
                  returned by the "time" function
10 State          State of node (e.g. IDLE, BUSY, DRAINING, DRAINED, DOWN)

Only the first item, Name, must be supplied in the configuration file.
Items two through eight can initially be read from the configuration file 
referred to by "NodeSpecConf" or can be established through communications 
with the Machine Status Daemon (MSD).
Items nine and higher are established through communications with other 
SLURM components, primarily the Machine Status Daemon (MSD).
The "Partition" specification will only come into play when SLURM starts to 
schedule resources and initiate jobs, in phase two of this project. 
If not otherwise specified, all nodes will be in partition zero.
If a node is not to be included in any partition, indicate this with the 
expression "Partition= ".
Change the value of MAX_PARTITION at SLURM build time to change the maximum 
partition value if so desired.
Lines in the configuration file having "#" in column one will be 
considered comments.
The configuration file should contain information about one node on 
a single line.
Each field should contain the field's name, an equal sign, and the value. 
Fields should be space or tab separated.
The default values for each node can be specified with a record in which 
"Name" is "DEFAULT". 
The default entry values will apply only to lines following it in the 
configuration file and the default values can be reset multiple times 
in the configuration file with multiple entries where "Name" is "DEFAULT".
Any node with less resources or a lower release level than specified 
will have the event logged and will not be configured for executing 
user jobs.
In order to support the concept of jobs requiring consecutive nodes, 
nodes should be place in this file in consecutive order.
The size of any field in the configuration file is limited to 1024 characters.
A sample NodeSpecConf file is included at the end of this document.

In order to simplify the building and support of SLURM, we will not 
incorporate a database into MSM at this time (if ever). 
The data can easily be kept in a C structure and written to a file 
for backup. 
This design also provides much greater speed than would be possible 
with a database.

The operation of MSM will be as follows:
 1. Read the SlurmConf file and confirm its integrity, log results
 2. Read the NodeSpecConf file and confirm its integrity, log results
 3. Contact the Machine Status Daemon for each of the CollectorNodes 
    to confirm each node is running. In the request, specify a 
    HeartBeatInterval of zero indicating daemon should report status 
    once and await further instructions.
 4. Divide the list of nodes which are *not* CollectorNodes into 
    lists of similar size with one list for each *responding* 
    CollectorNode. Save this mapping for fault-tolerance
 5. Send a message to the Machine Status Daemon on each of the 
    *responding* CollectorNodes with the configured HeartBeatInterval  
    and a node list. Each of these CollectorNodes will forward 
    the request to each of the listed nodes, collect all responses 
    along with its own, and reply in a single message to the MSM. 
 6. If a non-CollectorNode fails to respond within the HeartBeatTimeout 
    mark this node as "Down".
 7. If a CollectorNode fails to respond within the HeartBeatTimeout 
    mark this node as "Down", redistribute its list of nodes 
    among other CollectorNodes and resume from step five above.
 8. "Down" nodes should be tested for response periodically. The MSM 
    should test each for response on a "convenient" basis. 
9A. If a non-CollectorNode responds, add it to one of the CollectorNodes 
    lists and resume from step five above.
9B. If a CollectorNode responds, rebalance the CollectorNodes lists,
    and resume from step five above. Note that a Machine Status Daemon 
    must first be told to stop responding to one CollectorNode before 
    starting to respond to another one.

The MSM shall accept API requests on the MachStatusPort to set or get 
machine status information. 
Only user root will be permitted to set machine status information 
and authentication will be provided by virtue of the low port number. 
The most common set machine status operation is expected to be making 
a machine state "Up" after individual nodes are restarted. 
The API shall permit any user to get system information. 
It will be possible to get state information for individual machines 
or all machines. 
It will also be possible to get only specific fields on those nodes, 
for example "Get Name and State for all nodes". 
We do not expect to provide support for SQL queries, the filtering 
and other processing will be the responsibility of the application. 

We will provide a simple command-line interface to MSM utilizing 
the API described above. 
We anticipate providing this tool with support for: 
1. Identifying fields to be reported
2. Identifying the machines to have state reported
3. Sorting on specific fields
4. Filtering on specific field values (e.g. "State=DOWN")


Machine Status Daemon

The Machine Status Daemon (MSD) at this point in time will only 
confirm that the daemon is active and the node functioning. 
MSD will accept one argument, the HeartBeatInterval. 
A zero HeartBeatInterval indicates the program should 
report status once and await further instructions. 
A non-zero HeartBeatInterval indicates the program should 
report status, sleep for the specified interval, then repeat. 
CollectorNodes will be given a list of other nodes to 
correspond with. 
The MSD must open connections with MSDs on the specified 
nodes and forward the requests. 
The MSD collects responses as they arrive and combine them 
into a single message buffer. 
When all nodes have responded or when its HeartBeatInterval 
is exceeded by 50 percent, send the collected responses.


Notes

It is advisable to start the ControlMachine before any other 
of the cluster's nodes.

There is no necessity for synchronized clocks on the nodes 
(unlike LoadLeveler).

The hierarchical communications with CollectorNodes provide 
excellent scalability (unlike LoadLeveler).

I am assuming that all nodes will have the same daemons 
running with the exception of the ControlMachine.
The ControlMachine will direct each node to operate in a 
different fashion as needed.

Fault-tolerance will be built through mechanisms to save 
and restore the database using local and global file systems. 

Do we want to special case nodes going down by running some 
script that might send e-mail?

We need more complete documentation (e.g. man pages for all components).

Do we want to specify fully qualified names for all machines or 
specify a domain name for conciseness?

We need to discuss fault-tolerance, which requires the communications 
library design work.


Sample SlurmConf

# 
# Example SlurmConf
# Author: John Doe
# Date: 11/06/2001
#
ControlMachine    = lx_control
CollectorNodes    = lx01,lx02
NodeSpecConf      = /usr/local/SLURM/NodeSpecConf
PartitionConf     = /usr/local/SLURM/PartitionConf
#
MachStatusManager = /usr/local/SLURM/MachStatusManager
MachStatusDaemon  = /usr/local/SLURM/MachStatusDaemon
MachStatusPort    = 612
MachStatusDebug   = init,msg
#
HeartBeatInterval = 300
HeartBeatTimeout  = 600


Sample NodeSpecConf

# 
# Example NodeSpecConf
# Author: John Doe
# Date: 11/06/2001
#
Name=DEFAULT OS=Linux.2.4.7-10 CPUs=2 Speed=1.0 RealMemory=2048 VirtualMemory=4096 TmpDisk=16384 Partition=1
#
Name=lx01
Name=lx02
Name=lx03 Speed=1.5 RealMemory=3072 Partition=1,2
Name=lx04 CPUs=1 Speed=1.3 Partition=1,3
Name=lx05
Name=lx06


Dependencies

The Communications Library is required.


Module Testing

Test Machine Status Daemon with various time intervals including invalid values 
(see HeartBeatInterval).
Test Machine Status Daemon with various file location names including invalid names 
(see MachStatusDaemon).
Test Machine Status Daemon with various port numbers including invalid values 
(see MachStatusPort).
Test Machine Status Daemon with debug options including invalid values 
(see MachStatusDebug).
Review logs from above tests.



Integration and System Testing

Test Machine Status Manager fault-tolerance with nodes being dropped.
Test Machine Status Manager with and without CollectorNodes (various counts).
Test Machine Status Manager with various NodeSpecConf specifications 
(with and without various defaults).
Test Machine Status Manager from various file location names including invalid names
(see MachStatusManager).
Test Machine Status Manager with debug options including invalid values 
(see MachStatusDebug).
Test Machine Status Manager with various failure modes on both CollectorNodes 
and normal compute nodes.
Test Machine Status Manager API with all options both from a privileged port 
and a non-previlged port (non-root user).
Review logs from above tests.

Test Machine Status Tool, all options in various combinations per man pages.
