boot delay.

Code to determine if we need to reboot or not based on node_features

Bug 5308

Fixes bad casting caused by Tim on review.

Make one definitive copy of sort_part_tier() in partition_msg.c and use it.

Bug 3844, 5552.

and other drain + node state flags.

Bug 5505

Parameters=PreserveCaseUser in the slurmdbd.conf.

to cancel pending reboots on nodes.

Bug 5506

to control whether the Slurmd will shutdown itself down or not when a
reboot request is received.

Continuation of 54197d9e

Bug 5019

Bug 5505

Bug 3844.

Bug 5528.

Bug 5260

Only split nodes here if a node_features plugin is in use.

Otherwise node fragmentation will occur if the node config has
CPUs specified but not CoresPerSocket and Sockets.

This could be avoided by filling out the node definition, but
adding this workaround for backwards compatiblity.

Bug 5039.

Task prologs could set or modify this, so wait to create the
directory until after they've finished.

Bug 5367.

Bug 4373.

Bug 5481.

There are subtle issues involved in treating a TCP transmission
as a unidirectional message delivery layer.

The original code path looks like: connect(), write(), close().
But Linux handles the write() and close() asynchronously behind the
scenes, and does not block until that write() has been ACK'd by the
remote end. So the write() and close() may succeed, even with data
still in flight. A communication error - and message loss - would
have been silently ignored, leading to unreliable message transmission.

Worse yet, one side of the connection would believe it sent the message,
while the receive side swears it never saw the packets. This leads to
infrequent and yet seemingly impossible data loss, and a very tough
bug to chase down.

This teardown code tries to force the connection to shut down in an
orderly manner, giving Slurm a chance to catch a connection problem
and the upstream calling path an opportunity to retransmit.

This teardown code is based on an approach described in Section 7.5
of "UNIX Network Programming" Volume 1 (Third Edition), specifically
the subsection regarding SO_LINGER. (And also covers why SO_LINGER is
not sufficent to prevent this issue.)

Bug 5164.

Previously only a single task of a job array could preempt during
backfill scheduling. This allows multiple tasks to preempt and have
resources reserved in backfill.

Bug 5405.

The getopt format string needs to handle an option here,
and the --help output had not been corrected after 99b2c4e8.

Bug 5522.

for displaying unformtted timelimit numbers.

Bug 5407

Bug 5094

to dictate what state of node is after reboot.

Bug 5070

Bug 5468

This is a backport of commit cefc9ec1.

Bug 4918

scheduling cycle.  Primarily caused by EnforcePartLimits=ALL.

Bug 5452

Bug 5248.

Without Delegate=yes, systemd will "fix" the cgroup hierarchies whenever
'systemctl daemon-reload' is called, which will then remove any
restrictions placed on memory or device access for a given job.

This is a problem especially since 'systemctl daemon-reload' may be called
automatically by rpm/yum or a variety of config file mangers, leading to
jobs escaping from slurmd/slurmstepd's control.

This setting should work for systemd versions >= 205.
https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/

Bug 5292.

When a job with time_end=0 and TRES null exists from an association that is
currently inside a reservation, the hourly rollup segfaults.

Bug 5143

As reported by Avalon Johnson on slurm-users
https://groups.google.com/forum/#!topic/slurm-users/BsMQ7Uk1PLw
Bug 5287.

srun was already fixed in b7053bda (Bug 3294).

Bug 5126

Add salloc/sbatch/srun option of --gres-flags=disable-binding to disable
    filtering of CPUs with respect to generic resource locality. This option is
    currently required to use more CPUs than are bound to a GRES (i.e. if a GPU
    is bound to the CPUs on one socket, but resources on more than one socket
    are required to run the job). This option may permit a job to be allocated
    resources sooner than otherwise possible, but may result in lower job
    performance.
bug 5189

When printing arrays in squeue and setting the SLURM_BITSTR_LEN variable to 0
or to NULL, the length of the output defaulted to 64, when the documentation
says it will default to "unlimited". This patch fixes this situation.

Bug 5440

Documented, and code reads as needing, the node lock. But these
were incorrectly set as the job locks.

Bug 5394.

Needs the job write lock, as it may change job status not just node
status. Especially after commit 33e352a6.

Bug 5406.