where needed)

This reverts most of commit
https://github.com/SchedMD/slurm/commit/570941362ffdc57e9e3d4723bc4f728ae04789d8
and adds a call from slurmctld to srun prior to deallocating nodes and
notifying slurmd to cancel the tasks

Otherwise an aborted slurmstepd can cause the srun process to hang
indefinitely; a problem reported in trouble ticket 149.

If the slurmstepd connects task I/O, but aborts after srun accepts the connect
and before slurmstepd writes data then srun could possibly hand indefinitely.
This probably does not explain failures seen at CEA, but can't hurt matters.
then the sr

This could happen in srun too and would typically indicate that
the munged is too busy to respond

useless in most cases.

introduce step accounting for a Cray.

A dedicated thread (_kill_thr) is launched by slurmstepd at the end of a
step in order to destroy the IO thread if it does not manage to correctly
terminate by itself after 300 seconds.

Two bugs are corrected in this logic by this patch.

First, the performed sleep(300) is not protected against interruptions
and this delay can be reduced to a few seconds in case of signals received
by slurmstepd, thus, reducing the delay and forcing the IO thread to
terminate before the expiration of the grace time. The logic is modified
to ensure that the delay is respected using a loop around the sleep().

Second, to terminate the IO thread, a SIGKILL is delivered to the IO thread
using pthread_kill. However, sending SIGKILL using pthread_kill is a
process-wide operation (see man pthread_kill), thus all the slurmstepd
threads are killed and slurmstepd is terminated. This logic is modified
by using pthread_cancel() instead of pthread_kill() thus letting the
pthread_join() of _wait_for_io() having a chance to act as expected.

Without this patch, when _kill_thr is interrupted, slurmstepd is
terminated, letting the step in a incomplete state, as the node may not
have been able to send the REQUEST_STEP_COMPLETE to the controler.
Thus, consecutive steps can no longer be executed and stay permanently in
the "Job step creation temporarily disabled, retrying" state.

When requesting a particular nodelist for a step, if at least one of the node is
still used by a former step (no REQUEST_STEP_COMPLETE received from that node),
the current behavior is to return ESLURM_INVALID_TASK_MEMORY and srun aborting
with "Memory required by task is not available".

This can be reproduced by launching consecutive steps with the -w parameter set
to $SLURM_NODELIST and introducing delays in the spank epilog on the execution
nodes.

The behavior is changed to only defer the execution of the step by returning
ESLURM_NODES_BUSY when it is detected that some nodes are blocked because of
already used memory.

When using consecutive steps, it appears that in some cases, the time required
by the slurmstepd on the execution nodes to inform the controler of the completion
of the step is higher than the time required to request the following step.
In that scenario, the controler can reject the step by returning the error code
ESLURM_REQUESTED_NODE_CONFIG_UNAVAILABLE even if the step could be executed if
all the former steps were correctly finished.

This can be reproduced by launching consecutive steps and introducing dalys in
the spank epilog on the execution nodes.

The behavior is changed to only defer the execution of the step by returning
ESLURM_NODES_BUSY when all the available nodes are not idle considering the
former steps.

This is needed if the munge deamon is under very heavy load
(e.g. with 1000 slurmd daemons per compute node).

slurmctld restart.