Coverity 182357

before the primary takes control.

The primary will wait until the backup has replied with it all the way
done shutting down before it takes over.  Before it would just wait 2
seconds which probably was ok in most situations.  Now we don't return
until it is completely out.

This also ups the CONTROL_TIMEOUT from 10 -> 30 seconds.

TRES positions.

Bug 4665

Bug 4446

introduced in commit ea85d123

Bug 4613

Lost a "!" in the edit

CID 182336

Coverity CID 182335

Expand advanced reservation feature specification to support parenthesis and
    counts of nodes with specified features. Nodes with the feature currently
    active will be prefered.
bug 4604

Coverity CID 182262, 182263, 182264

Make sure that --batch option arguments are also found in job
  --constraint option.
bug 4604

Much of the logic does work, but better to prevent users from
trying this and failing in some way

Reported by Jenkins after 17.11 merge 93e67a13.

Commit 818a09e8 introduced a new state JOB_OOM and a new state reason
FAIL_OOM (OutOfMemory). The problem was that it based the decision upon
the value of the different memory.[*].failcnt being > 0.

That lead to "false positives" situations when the usage hit the limit
but the Kernel was able to reclaim pages and the process managed to finish
successfully. When this happens there might not necessary be OOM_KILL
events happening.

This patch makes it so the JOB_OOM state is set based upon OOM_KILL events
detected instead of usage hitting the limit. The usage hit will still
be logged as an info() message, and further work will be needed in the
master branch to better discern both type of events, maybe changing
the API and getting rid of the current SIG_OOM and a potential new
SIG_OOM_KILL.

OOM_KILL event is detected using the eventfd notification mechanism
on the cgroup v1 control/event files:
https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt

If we plan to support cgroup v2, we should monitor 'memory.events' file
modified events. That would mean that any of the available entries changed
its value upon notification.
Entries include: low, high, max, oom, oom_kill:
https://www.kernel.org/doc/Documentation/cgroup-v2.txt
https://patchwork.kernel.org/patch/9737381
but since this is a fairly recent change many sites might be running
kernels still not supporting this feature.

Bug 3820.

Correct SLURM_NTASKS and SLURM_NPROCS environment variable for heterogeneous
    job step. Report values representing full allocation.

Bug 4656.

This reverts commit d3141dc9.

Bug 4655

Turns out there are many ways to get this information directly from
the slurmstepd.  As you can already get this information from ps we
decided to just revert back to the old non-authenticated way of doing
things.

If we do need this in the future we need to patch the stepd as well as
the slurmd here in all the RPC's that try to grab this.

A user could easily run scontrol (or their own home baked thing)
on the node which will give them a direct contact with the slurmstepd.

This reverts commit c4fb9bc3.

This reverts commit d3141dc9.

Bug 4655

Turns out there are many ways to get this information directly from
the slurmstepd.  As you can already get this information from ps we
decided to just revert back to the old non-authenticated way of doing
things.

If we do need this in the future we need to patch the stepd as well as
the slurmd here in all the RPC's that try to grab this.

A user could easily run scontrol (or their own home baked thing)
on the node which will give them a direct contact with the slurmstepd.

Bug 4656

Introduced in commit f2efbb60
Coverity CID 182262, 182263 and 182264

bug 4607