burst_buffer/cray - Do not execute "pre_run" operation until after all nodes
    are booted and ready for use.
bug 3461

partitions.

Insure job does not start running before node is booted and PrologSlurmctld
    is complete.
bug 3446

Ported from 7a4aa7f2.

burst_buffer/cray - Support default pool which is not the first pool
    reported by DataWarp and log in Slurm when pools that are added or removed
    from DataWarp.
bug 3453

This reverts commit fd690a9c.

was there for jobs and steps, but for no real reason.  I couldn't see a
reason anyway.

Jobs preempted with PreemptMode=REQUEUE were incorrectly recorded as
REQUEUED in the accounting.

Bug 3444

bug 3448

i.e.

AcctGatherInterconnectType = ofed,none

Bug 3412

Actual code change to make it happen.  Just renaming things, nothing
really changed.

Bug 3447

Makes it so that an srun/salloc initiated from a cluster with a one
switch type will work on a different cluster with a different switch
type.

dynamically or statically to libslurm.

This can result in much smaller binaries, but isn't the easiest to develop against as if the files in
src/common or src/*api change and the binaries linking against them are compiled directly
libslurm.so doesn't get recompiled automatically, you have to manually do it.  But in production
this could give large benefits as Slurm's footprint is now considerably smaller.

This used to not work on environments like AIX where plugins couldn't resolve variables used in
the parent program, but this appears to not be the case now we link to the "full" .so which exports
everything as the .o does.

Bug 3444

Fix srun I/O race condtion to eliminate a error message that might be
    generated if the application exits with outstanding stdin.
bug 3166

Properly set SLURM_JOB_GPUS environment variable for Prolog.
bug 3437

Clear job's reason of "BeginTime" in a more timely fashion and/or prevents
    them from being stuck in a PENDING state. There are multiple ways of
    clearing the reason, especially on a lightly loaded system, but the
    state can persist indefinitely on a heavily loaded system.
bug 3368

Fix to logic for getting expected start time of existing job ID with
    explicit begin time that is in the past. Previous logic would
    compare that (past) begin time with advanced reservations that
    would compete with it rather than the current time.

CRAY systems only: TaskPlugins must list task/cgroup before task/cray in
    order for the cgroup files to be created before task/cray runs. Without
    this change, the task/cray plugin frequently produces errors about the
    "mems" file being missing. The errors don't seem consistent, so this
    probably involves a race condition. Note that NERSC uses this order
    today and I changed read_config.c to produce a fatal error if the order
    is reversed.

Turns out this never worked, ever.  What used to happen is if the protocol_version that was
read in didn't match the rpc_version given to unpack things was just 0.  What this does
now is set the rpc_version to what was stored making it all good.

Revert logic originally added for bug 3166.
  Revisit as time permits.
bug 3166

Interpet all format options in output/error file to log prolog errors. Prior
    logic only supported "%j" (job ID) option.
bug 3354

Bug 3431

burst_buffer/cray - Fix race condition that could cause multiple batch job
    launch requests resulting in downed nodes.
bug 3366

This reverts commit b9bff82f.