from_tres_rec was being added to dest_tres_list and being free'd twice
when free'ing from_tres_list and dest_tres_list.

Continuation of a4a91719

Bug 5260

Only split nodes here if a node_features plugin is in use.

Otherwise node fragmentation will occur if the node config has
CPUs specified but not CoresPerSocket and Sockets.

This could be avoided by filling out the node definition, but
adding this workaround for backwards compatiblity.

Bug 5039.

Task prologs could set or modify this, so wait to create the
directory until after they've finished.

Bug 5367.

Bug 4373.

Bug 4373.

Which can be used to force to retrieve the information about all the
components in a heterogeneous job or just the ones selected, leaving
untouched the way regular jobs and/or arrays are retrieved.

Bug 4373.

cluster_name is now passed as an argument, since it'll be needed
for a subsequent commit subquery. No functional change.

Bug 4373.

No functional change.

Bug 4373.

No functional change.

Bug 4373.

Bug 5481.

for an allocation.

Bug 5293

when validating memory limits

Continuation of b502d179

After changes to slurm_send_only_node_msg(), this message is much
more likely to appear on systems with overloaded interconnects since
that connection handling code may end up retransmitting messages
that were actually received (but that the transmit side could not
verify were delivered successfully).

As the error() message stated, this isn't actually an error, and
the code will proceed happily past this point. So drop the debug
level, and remove the surrealist "this is not an error" part.

Bug 5164.

There are subtle issues involved in treating a TCP transmission
as a unidirectional message delivery layer.

The original code path looks like: connect(), write(), close().
But Linux handles the write() and close() asynchronously behind the
scenes, and does not block until that write() has been ACK'd by the
remote end. So the write() and close() may succeed, even with data
still in flight. A communication error - and message loss - would
have been silently ignored, leading to unreliable message transmission.

Worse yet, one side of the connection would believe it sent the message,
while the receive side swears it never saw the packets. This leads to
infrequent and yet seemingly impossible data loss, and a very tough
bug to chase down.

This teardown code tries to force the connection to shut down in an
orderly manner, giving Slurm a chance to catch a connection problem
and the upstream calling path an opportunity to retransmit.

This teardown code is based on an approach described in Section 7.5
of "UNIX Network Programming" Volume 1 (Third Edition), specifically
the subsection regarding SO_LINGER. (And also covers why SO_LINGER is
not sufficent to prevent this issue.)

Bug 5164.

Bug 5164.

Do not use slurm_send_only_node_msg().

There is no way to tell if the srun has received the message before
the socket is shutdown if we do not wait to receive data. Use
slurm_send_recv_rc_msg_only_one() instead, and send back a response
from the other side.

We still need the older (and unreliable) behavior when talking to older
srun client commands, so make this change dependent on the protocol_version
field in the message.

Bug 5164.

Bug 5164.

Previously only a single task of a job array could preempt during
backfill scheduling. This allows multiple tasks to preempt and have
resources reserved in backfill.

Bug 5405.

This appears to be an oversight of 865338c7 where the cont_id check
was changed from NO_VAL64 to INFINITE64. The cont_id is initialized to
NO_VAL64 in src/common/slurm_jobacct_gather.c.

The getopt format string needs to handle an option here,
and the --help output had not been corrected after 99b2c4e8.

Bug 5522.

Bug 5527.

for displaying unformtted timelimit numbers.

Bug 5407

Bug 5523

Bug 5523

Update slurm.spec and slurm.spec-legacy as well.

CID 187301

CID 187302

Bug 5308