If a job --mem-per-cpu limit exceeds the partition or system limit, then
scale the job's memory limit and CPUs per task to satisfy the limit.
bug 848

Properly enforce job --requeue and --norequeue options. Previous
logic was in three places not doing so (either ignoring the value,
ANDing it with the JobRequeue configuration option or using the
JobRequeue configuration option by itself).
bug 821

Also correct related documentation

Completing nodes is removed when calling _try_sched() for a job, which
is the case in select_nodes(). If _try_sched() thinks the job can run
now but select_nodes() returns ESLURM_NODES_BUSY, the backfill loop will
be ended.

Correct sinfo --sort fields to match documentation: E => Reason,
H -> Reason Time (new), R -> Partition Name, u/U -> Reason user (new)

Related to bug 771

cnode counts.

Switch/nrt - Properly track usage of CAU and RDMA resources with multiple
tasks per compute node. Previous logic would allocate resources once per
task and then deallocate once per node, leaking CMA and RDMA resources
and preventing their use by future jobs.

On switch resource allocation failure, free partial allocation.
Failure mode was CAU could be allocated on some nodes, but not
others. The CAU allocated on nodes and switches up to the failure
point were never released.

Fix logic bugs for SchedulerParameters option of max_rpc_cnt.
Scheduling would be delayed for job arrays and
backfill scheduling would be disabled unless max_rpc_cnt > 0.

on Mixed state.

This largely reverts commit 0ec2af27
just to cut down on some logging

in it.

Signed-off-by: Danny Auble <da@schedmd.com>

Rather than treat invalid SchedulerParameters options as a fatal
error, print an error and use to the default value.

Disables job scheduling when there are too many pending RPCs

This is related to defering batch job scheduling if there
are a bunch of requests pending

rather than 1 sec interval retries

If pthread_create call fails, decrease sleep before retry from
1 sec to 0.1 sec

This also reverts commit 8cff3b08 and
ced2fa3f