There are subtle issues involved in treating a TCP transmission
as a unidirectional message delivery layer.

The original code path looks like: connect(), write(), close().
But Linux handles the write() and close() asynchronously behind the
scenes, and does not block until that write() has been ACK'd by the
remote end. So the write() and close() may succeed, even with data
still in flight. A communication error - and message loss - would
have been silently ignored, leading to unreliable message transmission.

Worse yet, one side of the connection would believe it sent the message,
while the receive side swears it never saw the packets. This leads to
infrequent and yet seemingly impossible data loss, and a very tough
bug to chase down.

This teardown code tries to force the connection to shut down in an
orderly manner, giving Slurm a chance to catch a connection problem
and the upstream calling path an opportunity to retransmit.

This teardown code is based on an approach described in Section 7.5
of "UNIX Network Programming" Volume 1 (Third Edition), specifically
the subsection regarding SO_LINGER. (And also covers why SO_LINGER is
not sufficent to prevent this issue.)

Bug 5164.

The getopt format string needs to handle an option here,
and the --help output had not been corrected after 99b2c4e8.

Bug 5522.

Bug 5070

Bug 5468

This is a backport of commit cefc9ec1.

scheduling cycle.  Primarily caused by EnforcePartLimits=ALL.

Bug 5452

Without Delegate=yes, systemd will "fix" the cgroup hierarchies whenever
'systemctl daemon-reload' is called, which will then remove any
restrictions placed on memory or device access for a given job.

This is a problem especially since 'systemctl daemon-reload' may be called
automatically by rpm/yum or a variety of config file mangers, leading to
jobs escaping from slurmd/slurmstepd's control.

This setting should work for systemd versions >= 205.
https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/

Bug 5292.

When a job with time_end=0 and TRES null exists from an association that is
currently inside a reservation, the hourly rollup segfaults.

Bug 5143

As reported by Avalon Johnson on slurm-users
https://groups.google.com/forum/#!topic/slurm-users/BsMQ7Uk1PLw
Bug 5287.

srun was already fixed in b7053bda (Bug 3294).

Bug 5126

When printing arrays in squeue and setting the SLURM_BITSTR_LEN variable to 0
or to NULL, the length of the output defaulted to 64, when the documentation
says it will default to "unlimited". This patch fixes this situation.

Bug 5440

Documented, and code reads as needing, the node lock. But these
were incorrectly set as the job locks.

Bug 5394.

Needs the job write lock, as it may change job status not just node
status. Especially after commit 33e352a6.

Bug 5406.

Add errno to info message in the SlurmDBD log, and pass the actual
errno back to the sacctmgr process so the user can see it.

Bug 5152.

- avoid `abort()` when collective is failed
- added logging of coll details for fail cases

Bug 5067

Note, this is setting it up so we can use defunct functions.  It will
probably need to be properly fixed in a future version so we don't
do this.

with preemption or when job requests a specific list of hosts.

Bug 5293.

Continuation of 923c9b37.

There is a delay in the cgroup system when moving a PID from one cgroup
to another. It is usually short, but if we don't wait for the PID to
move before removing cgroup directories the PID previously belonged to,
we could leak cgroups. This was previously fixed in the cpuset and
devices subsystems. This uses the same logic to fix the freezer
subsystem.

Bug 5082.

So that multiple nodes changes will be reported on one line rather than one
line per node. Otherwise this could lead to performance issues when reloading
slurmctld in big systems.

Bug4980

Currently, no caller checks the return code.

Bug 5164

Some job fields can change in the course of scheduling. This patch
reinitializes previously adjusted job fields to their original value
when validating the job memory in multi-partition requests.

Bug 4895.

This reverts commit bf4cb0b1.

Bug 5240, Bug 4895 and Bug 4976.

When one asks for an inactive feature and also specifies the node with -w flag,
the node will be rebooted despite it may contain running jobs.

bug4821

and avoid race condition calling task before proctrack can introduce.

Bug 5319

delayed until the first job completes execution and it's burst buffer
stage-out is completed.

Bug 4675

Bug 5159.

Previously the function was only testing against the first partition in
the job_record. Now it detects if the job request is multi partition and
if so then loops through all of them until the job will run in any or
until the end of the list, returning the error code from the last one if
the job won't run in any partition.

Bug 5185

When requesting specific jobids with sacct, the starttime of the request
is 0, which will cause the time range to be outside of the
MaxQueryTimeRange range -- if specified. When requesting specific
jobids, sacct should be able to find the job whenever it started --
unless confined to a smaller range with -S and/or -E.

Bug 5009

Specifically due to SELECT ... FOR UPDATE ones.

Bug 5086.

Bug 5270.

Instead of unintentionally rejecting the update from a non-Administrator
if the job_submit plugin modified that field.

Bug 5306.

Bug 5286

RHEL 6 (and related) use lua as the package name, test
if that package exists with a version >= 5.1 if the other
tests have already failed.

Bug 5263.

And do not list each individual sensor reading but just the computed
sum of each one grouped by key.

Bug5274

This is in anticipation of an upcoming change to the cgroup hierarchy
on a future CLE release.

Bug 5145.