Bug 1627

This is a special case. This change documents the way Slurm has
always worked.

Make sure that old step data is purged when a job is requeued.
Without this logic, if a job terminates abnormally then old step
data may be left in slurmctld. If the job is then requeued and
started on a different node, referencing that old job step data
can result in abnormal events. One specific failure mode is if
the job is requeued on a node with a different number of cores,
and the step terminated RPC arrives later, the job and step
bitmaps of allocated cores can differ in size generating an
abort.
bug 1660

Bug 1618

cpu count.

random crashing in db2 when the slurmctld is exiting.

Signed-off-by: Danny Auble <da@schedmd.com>

In slurmctld communication agent, make the thread timeout be the configured
value of MessageTimeout (or 30 seconds, whichever is larger) rather than
30 seconds.

Fix scancel bug which could return an error on attempt to signal a job step.
A simple "scancel 12.3" to signal a specific job step would fail. Adding
another option (say "-i", "--partion=", etc.) would fix this.

Modify slurmctld's parsing of a job_id string for the job_signal and
job_requeue calls to treat a job ID value of "#_*" as representing
all tasks in a job ID number "#". Previously treated as invalid input.

Also set the last_job_update time so that if a pending job is killed,
then that is reported immediately by "squeue -i#" (previously it
may keep reporting stale date.

Trying to avoid having technical questions sent to "sales@schedmd.com"

This avoids letting the queued scheduling thread from starting
if the main scheduling loop is still running.

This reverts commit f9ebf5ad.

Conflicts:
	src/plugins/select/alps/basil_interface.c

before ending the job.

will make it so the slurmctld will not signal the apid's in a batch job.
Instead it relies on the rpc coming from the slurmctld to kill the job to
end things correctly.

Make this be the minimum time between the end of one scheduling
cycle and the start of the next cycle (rather than using start
times for both).
Set the default value to 1,000,000 microseconds for Cray/ALPS
systems.

Refactor scancel so that all pending jobs are cancelled before starting
cancellation of running jobs. Otherwise they happen in parallel and the
pending jobs can be scheduled on resources as the running jobs are being
cancelled.