burst_buffer/cray - Do not execute "pre_run" operation until after all nodes
    are booted and ready for use.
bug 3461

Insure job does not start running before node is booted and PrologSlurmctld
    is complete.
bug 3446

Ported from 7a4aa7f2.

burst_buffer/cray - Support default pool which is not the first pool
    reported by DataWarp and log in Slurm when pools that are added or removed
    from DataWarp.
bug 3453

This reverts commit fd690a9c.

Jobs preempted with PreemptMode=REQUEUE were incorrectly recorded as
REQUEUED in the accounting.

Bug 3444

bug 3448

Bug 3447

Bug 3444

Clear job's reason of "BeginTime" in a more timely fashion and/or prevents
    them from being stuck in a PENDING state. There are multiple ways of
    clearing the reason, especially on a lightly loaded system, but the
    state can persist indefinitely on a heavily loaded system.
bug 3368

Fix to logic for getting expected start time of existing job ID with
    explicit begin time that is in the past. Previous logic would
    compare that (past) begin time with advanced reservations that
    would compete with it rather than the current time.

Turns out this never worked, ever.  What used to happen is if the protocol_version that was
read in didn't match the rpc_version given to unpack things was just 0.  What this does
now is set the rpc_version to what was stored making it all good.

Bug 3431

burst_buffer/cray - Fix race condition that could cause multiple batch job
    launch requests resulting in downed nodes.
bug 3366

This reverts commit b9bff82f.

slurmctld/agent race condition fix: Prevent job launch while PrologSlurmctld
    daemon is running or node boot in progress.
bug 3366

If a lower version client would try to communicate with a higher version
controller the dbd would return the controller's version and the client
would use that version to talk to the controller. When the controller
would respond, the client wouldn't know how to unpack the higher version
msg.

Bug 3398

bug 3099

This reverts commit e92b49d3.

instead of also in the backfill scheduler.

Bug 3405.

acct_policy_job_runnable_pre_select() calls assoc_mgr_set_qos_tres_cnt()
without tres READ_LOCK.

Note that existing code does not modify the tres structures, so this
cannot currently lead to a race condition.

Bug 3406.

job_submit/cnode was previously removed by commit 63bc71ed.

Bug 3403.

burst_buffer/cray - Avoid "pre_run" operation if not using buffer (i.e.
    just creating or deleting a persistent burst buffer).
bug 3391

Previous job state information was "PENDING" rather than "REQUEUED"
  for each job requeued due to a burst buffer error.
bug 3388

scheduling a Datawarp job.

The assoc_mgr lock needs to happen before the bb_state.bb_mutex.  One place
this could cause deadlock is from src/slurmctld/controller.c
_accounting_cluster_ready() which calls clusteracct_storage_g_cluster_tres
which inturn calls bb_g_job_set_tres_cnt which calls bb_p_job_set_tres_cnt
which will lock the bb_muxtex after the assoc_mgr is already locked.

Bug 3389

Cache results of bit_set_count() calls.

Bug 3393.