Bug 3267.

Termination can race against step creation if, e.g., ill-behaved SPANK plugins
are in use.

Bug 3248.

sched/backfill plugin: Make malloc match data type (defined as uint32_t and
    allocated as int). No failures observed, if type "int" is smaller than
    "uint32_t", it could result in an invalid memory reference.

Fix API call: slurm_job_cpus_allocated_str_on_node_id() and
in turn slurm_job_cpus_allocated_str_on_node() to return correct
results for anything but first node. This was caused by missed logic
to calculate fist bit belongs to particular node. Lookup was always
starting from bit 0.

Bug 3266.

After one second of wall time, simulate the termination of all remaining
   running jobs in order to respond in a reasonable time frame.
bug 3275

Modify backfill algorithm to improve performance with large numbers of
    running jobs. Group running jobs that end in a "similar" time frame using a
    time window that grows exponentially rather than linearly. The original
    window sizes were (in units of minutes):
    0, 1, 2, 3, 4, 5, 6, 7, ... minutes
    The new window sizes are:
    0.5, 1, 2, 4, 8, 16, 32, ... minutes
    This can dramatically reduce the number of instances where the very time
    consuming "can the pending job run now" operation is executed, especailly
    if there are 1000+ running jobs.
bug 3275

If a node is booting for some job, don't allocate additional jobs to the
    node until the boot completes.
but 3256

Found with valgrind. Bug 2846.

Move where we set the configuration table bitmaps in order to support
  the backup slurmctld starting and recovering previously saved
  KNL mode information (which can necessitate rebuilding the node
  configuration table).
bug 3241

Bug 3255.

Bug 3253.

If the input value mod 512 == 0, the value would be subject to
unintended rounding. Rework the function to check against this
on each unit promotion.

Bug 3252.

It was causing the loss of node available_features on startup with
  node_features/knl_cray
bug 3241

Set per-node HBM availability as a GRES based upon the KNL node's
  MCDRAM state
bug 3171

Caused by race for local_energy which is dynamically allocated. Bail out
of the update if that hasn't been allocated yet.

Bug 3237.

bug 3213

select/linear plugin modified to better support heterogeneous clusters when
    topology/none is also configured. Note that use of the select/cons_res
    plugin is strongly recommended for heterogeneous clusters. The use of
    OverSubscribe=exclusive can be used if whole node allocations is
    desired.
bug 3212

Bug 3224.

If a job is started by the main scheduling logic and requeued while
  the backfill scheduler has locks released, that can result in an
  invalid data structure in select/cons_res. Namely, the backfill
  scheduler's attempt to start the job would clear the job resources
  node_bitmap. That leaves a NULL pointer in the select/cons_res
  plugin generating an abort. (That pointer is needed to clean up
  the job allocation records when the Epilog or Cray Node Health
  Check, NHC, are complete and the resources become available for
  another job.
bug 3230

Backup slurmctld will now
1. Not abort due to NULL pointer (needed to move code around on restart)
2. Recover KNL MCDRAM and NUMA modes from state save files if capmc and
   cnselect not available
bug 3241

cray/burst_buffer - Update "instance" parsing to match updated dw_wlm_cli
    output.
bug 3222

cray/burst_buffer - Preserve job ID and don't translate to job array ID
  after slurmctld restart. Prior logic would not set array_task_id to
  NO_VAL, so all job-buffer IDs would be reported in the form
  "JobID=0_0(123)" rather than "JobID=123"

cray/busrt_buffer - Internally track both allocated and unusable space.
    The reported UsedSpace in a pool is now the allocated space (previously was
    unusable space). Base available space on whichever value leaves least free
    space.
bug 3222

and request --ntasks-per-core=1 and only 1 task on the node
the slurmd would abort on an infinite loop fatal.

Regression is from commit 5265420d.

Without this fix you can get into an infinite loop in the task/affinity
plugin.  The loop is handled by producing a fatal.

Bug 3118

cray/busrt_buffer - Fix for double counting of used_space at slurmctld
    startup.
bug 3222

cray/busrt_buffer - If total_space in a pool decreases, reset used_space
    rather than trying to account for buffer allocations in progress.
bug 3222

more time than should be allowed would be accounted for.

This only happened on jobs in the completing state when the slurmctld
was shutdown.

This will also be enhanced in 17.02 as the job's end_time_exp is not
stored which is needed to determine if the job has already been through
the decay_thread at end of job.

Bug 3162

issue with gang scheduling.

Bug 3211

Fix bug that was clearing MAINT mode on nodes scheduled for reboot (bug
    introduced in version 16.05.5 to address bug in overlapping reservations,
    commit 5eee1d28). Note that a node's
    MAINT flag is used for both a requested reboot and maintenance reservation.
    What I'd like to do is add a new node state flag to differenciate
    between these two cases, but that involves some significant changes
    that could introduce instability, so it will be defered to version
    17.02
bug 3210

requested with -n tasks < hosts from -w hostlist.

bug 2149

Bug 3194

Document that node Weight takes precedence over load with LLN scheduling.
bug 3204

task/cray's _get_numa_nodes() function needs to run before task/cgroup
cleans up the cgroup hierarchies, otherwise ALPS memory compaction will
never run.

Also move task_p_add_pid() outside the #ifdef HAVE_NATIVE_CRAY
block so that the plugin will load (albeit without any functionality)
on non-Cray systems for testing purposes.

Revise documentation and provided slurm.conf templates as well.

Bug 3154.