this happens anywhere in the code but just incase it ever does, lets fix
it.

This can be caused by a core reservation on nodes which get taken
  out of the system or fail.
bug 2296

salloc/sbatch/srun did not mention this. Also reference OverTimeLimit
as another option affecting the final run time. Bug 2309.

the job starts update the cpus_per_task appropriately.

This also moves update num_tasks to after the setting of node counts on
an update.  It didn't appear to matter, but the cpus_per_task and
pn_min_cpus had to be figured out after the cpus and nodes were set but
before tasks.

Bug 2302

burst_buffer/cray - Improve tracking of allocated resources to handle race
    condition when reading state while buffer allocation is in progress.
Also initialize a mutex

DBD for the first time.  The corruption is only noticed at shutdown.

Bug 2293

Set job's reason to "Priority" when higher priority job in that partition
    (or reservation) can not start rather than leaving the reason set to
    "Resources".
bug 2285

error message.

If a reservation's nodes value is "all" then track the current nodes in the
    system, even if those nodes change. Nodes will automatically be added to
    or removed from a reservation when slurm.conf changes.
bug 2204

Bug 2281

Rename the variable to match rest of codebase while here. This is related to
bug 2295, although snprintf() protects against buffer overflow in 15.08 and up.

Later releases have switched over to snprintf to avoid this issue, but 14.11
did not get that patch. Bug 2295.

time.

static/overlap systems when some hardware issue happens when restarting
the slurmctld.

a dynamic system and mark the block in error on a static/overlap system.

Bug 2273

Don't use lower weight nodes for job allocation when topology/tree used.
bug 2284

Preemption/gang scheduling: If a job is suspended at slurmctld restart or
    reconfiguration time, then leave it suspended rather than resume+suspend.
bug 2274

task/affinity: Disable core-level task binding if more CPUs required than
    available cores.
bug 2267

Log as error if more than 3 aeld connects per second that cause is
  likely duplicate slurmctld daemon
bug 2278

This is needed to properly enforce limits and account for usage.

Remove the 1024-character limit on lines in batch scripts, which was
causing long lines to be silently truncated. I noticed it when jobs were
getting created with fewer dependencies than specified.

Also increase the line length when showing job info.

sched/backfill: If a job can not be started within the configured
    backfill_window, set it's start time to 0 (unknown) rather than the end
    of the backfill_window.
bug 2100

If a pending job array has multiple reasons for being in a pending state,
    then print all reasons in a comma separated list.

Before:
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
       97354_[1-4]     debug      tmp    jette PD       0:00      1 (Resources)

After:
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
       97354_[1-4]     debug      tmp    jette PD       0:00      1 (Resources,JobHeldUser)

Bug 2130

Move slurmctld mail handler to separate thread for improved performance.
Original logic did fork/exec without separate thread and if the
slurmctld memory size is huge, then the time required for fork()
to complete can be significant.
bug 2252

Avoid hard-link/copy of script/environment files for job arrays. Use the
    master job record file for all tasks of the job array.
back port of commit 41eead64
bug 2220

Refactor logic to synchronize active bach jobs and their script/environment
    files, reducing overhead dramatically for large numbers of active jobs.
backport of commit 41eead64
bug 2220

If a job can never be started due to unsatisfied job dependencies, report
    the full original job dependency specification rather than the dependencies
    remaining to be satisfied (typically NULL).
bug 2197

in use.

_slurm_rpc_complete_job_allocation.

This is a rewrite of 438365ec which didn't catch the job_ptr wasn't
in a lock so the memory issue could still of existed.  This hopefully fixes
all the spots the job_ptr wasn't in the lock.

Fixes bug 2146

Bug 2261

15.08.5.

Bug 2243