Added SchedulingParameters option of "bf_job_part_count_reserve". Jobs below
    the specified threshold will not have resources reserved for them.
bug 3275

messages.  Hopefully this will reduce the number of messages lost when
filling up memory when the database/DBD is down.

sched/backfill plugin: Make malloc match data type (defined as uint32_t and
    allocated as int). No failures observed, if type "int" is smaller than
    "uint32_t", it could result in an invalid memory reference.

Fix API call: slurm_job_cpus_allocated_str_on_node_id() and
in turn slurm_job_cpus_allocated_str_on_node() to return correct
results for anything but first node. This was caused by missed logic
to calculate fist bit belongs to particular node. Lookup was always
starting from bit 0.

Bug 3266.

After one second of wall time, simulate the termination of all remaining
   running jobs in order to respond in a reasonable time frame.
bug 3275

Modify backfill algorithm to improve performance with large numbers of
    running jobs. Group running jobs that end in a "similar" time frame using a
    time window that grows exponentially rather than linearly. The original
    window sizes were (in units of minutes):
    0, 1, 2, 3, 4, 5, 6, 7, ... minutes
    The new window sizes are:
    0.5, 1, 2, 4, 8, 16, 32, ... minutes
    This can dramatically reduce the number of instances where the very time
    consuming "can the pending job run now" operation is executed, especailly
    if there are 1000+ running jobs.
bug 3275

Modify several NEWS items for greater clarity.

Add new PriorityFlags option of INCR_ONLY, which prevents a job's priority
    from being decremented.
Added for NERSC

If a node is booting for some job, don't allocate additional jobs to the
    node until the boot completes.
but 3256

Found with valgrind. Bug 2846.

Move where we set the configuration table bitmaps in order to support
  the backup slurmctld starting and recovering previously saved
  KNL mode information (which can necessitate rebuilding the node
  configuration table).
bug 3241

Bug 3255.

Since CLE 5.1 I believe, /var/spool/alps is a symlink to
/var/opt/cray/alps/spool, so it should be safe to use
/var/opt/cray/alps/spool.

Bug 3253.

Bug 3253.

If the input value mod 512 == 0, the value would be subject to
unintended rounding. Rework the function to check against this
on each unit promotion.

Bug 3252.

It was causing the loss of node available_features on startup with
  node_features/knl_cray
bug 3241

Check for zonesort file first, to save time over attempting to load
  a module that is already loaded. It may be loaded by default per
  administrator configuration.

Set per-node HBM availability as a GRES based upon the KNL node's
  MCDRAM state
bug 3171

Caused by race for local_energy which is dynamically allocated. Bail out
of the update if that hasn't been allocated yet.

Bug 3237.

Add new node state flag of NODE_STATE_REBOOT for node reboots triggered by
    "scontrol reboot" commands. Previous logic re-used NODE_STATE_MAINT flag,
    which could lead to inconsistencies. Add "ASAP" option to "scontrol reboot"
    command that will drain a node in order to reboot it as soon as possible,
    then return it to service.
bug 3210

bug 3213

select/linear plugin modified to better support heterogeneous clusters when
    topology/none is also configured. Note that use of the select/cons_res
    plugin is strongly recommended for heterogeneous clusters. The use of
    OverSubscribe=exclusive can be used if whole node allocations is
    desired.
bug 3212

Bug 3224.

If a job is started by the main scheduling logic and requeued while
  the backfill scheduler has locks released, that can result in an
  invalid data structure in select/cons_res. Namely, the backfill
  scheduler's attempt to start the job would clear the job resources
  node_bitmap. That leaves a NULL pointer in the select/cons_res
  plugin generating an abort. (That pointer is needed to clean up
  the job allocation records when the Epilog or Cray Node Health
  Check, NHC, are complete and the resources become available for
  another job.
bug 3230

Backup slurmctld will now
1. Not abort due to NULL pointer (needed to move code around on restart)
2. Recover KNL MCDRAM and NUMA modes from state save files if capmc and
   cnselect not available
bug 3241

cray/burst_buffer - Update "instance" parsing to match updated dw_wlm_cli
    output.
bug 3222

Add "FreeSpace" information for each pool to the "scontrol show burstbuffer"
    output. Required changes to the burst_buffer_info_t data structure.
bug 3222

cray/burst_buffer - Preserve job ID and don't translate to job array ID
  after slurmctld restart. Prior logic would not set array_task_id to
  NO_VAL, so all job-buffer IDs would be reported in the form
  "JobID=0_0(123)" rather than "JobID=123"

cray/busrt_buffer - Internally track both allocated and unusable space.
    The reported UsedSpace in a pool is now the allocated space (previously was
    unusable space). Base available space on whichever value leaves least free
    space.
bug 3222

Add LaunchParameters=mem_sort option to set configur running of zonesort
    by default at step startup.
Also add documentation about zonesort on KNL web page
bug 3188

and request --ntasks-per-core=1 and only 1 task on the node
the slurmd would abort on an infinite loop fatal.

Regression is from commit 5265420d.

Without this fix you can get into an infinite loop in the task/affinity
plugin.  The loop is handled by producing a fatal.

Bug 3118

cray/busrt_buffer - Fix for double counting of used_space at slurmctld
    startup.
bug 3222

cray/busrt_buffer - If total_space in a pool decreases, reset used_space
    rather than trying to account for buffer allocations in progress.
bug 3222

bug 3188

more time than should be allowed would be accounted for.

This only happened on jobs in the completing state when the slurmctld
was shutdown.

This will also be enhanced in 17.02 as the job's end_time_exp is not
stored which is needed to determine if the job has already been through
the decay_thread at end of job.

Bug 3162

bug 3139