This bug was introduced by commit 21c52d2f
which fixed a different problem tracking resources associated with suspended
jobs. There are subtle differences between jobs that are suspended by a
user/administrator and jobs suspended by gang scheduling which resulted in
undercounting allocated CPUs when a job suspended by gang scheduling
was active at the same time of a slurmctld reconfiguration request.
See bugs 2353 (original bug related to commit 21c52d2f
and bug 2765

accounts) no default account is printed, previously NULL was printed.

This is just not printing it, but whole function should probably be
revisited as the rigmarole can probably be avoided as we always know what
the default is going to be if none is specified (first off the list).

The problem with that though is if the user has been added to a cluster
already and they have a default, but then added to a new cluster where
they don't have a default.  In this case you want to keep the first
clusters default, but set the default for the second cluster.

Bug 2725

Add missing unlock before return. Coverity 44888.

Coverity 44891.

Coverity 44992.

Needs to unlock here, not re-lock the lock.

Prevent '--preserve' from being inadvertanly enabled by '-j'.

Still testing 16.05 on my NetBSD/amd64 workstation ...
Just encountered a crash with scancel(1).
njoly@lanfeust [~]> sbatch --wrap "sleep 3600"
Submitted batch job 4680
njoly@lanfeust [~]> scancel 4680
scancel: Error detected by libpthread: Invalid condition variable.
Detected by file "/local/src/NetBSD/src/lib/libpthread/pthread_cond.c", line 140, function "pthread_cond_timedwait".
See pthread(3) for information.
zsh: abort (core dumped)  scancel 4680
Checking the code show indeed that pthread_cond_wait() call from scancel.c:_signal_job_by_str() use an uninitialised condition variable "num_active_threads_cond"
The attached patch, which add the missing pthread_cond_init() seems to fix it.
bug 2753

and the slurmctld doesn't wait long enough for the response it would give
up leaving the connection open and create a situation where the next message
sent could receive the response of the first one.

Bug 2739

Bug #2713.

Correct description of the SLURMD_NODENAME environment variable
in the sbatch and srun man pages.

when in use.

The problem here is the polling threads in the various acct_gather codes
were detached and could possibly still be polling after the plugin had
been unloaded making a seg fault with a backtrace like this...

#0  0x00007fe7af008c00 in ?? ()
#1  0x00007fe7b1138479 in __nptl_deallocate_tsd () at pthread_create.c:175
#2  0x00007fe7b11398b0 in __nptl_deallocate_tsd () at pthread_create.c:326
#3  start_thread (arg=0x7fe7b1f12700) at pthread_create.c:346
#4  0x00007fe7b0e6fb5d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

The fix was to make the threads non-detached and join them before calling
a dlclose.

trying to verify the cluster name (which may try to /create/ files or
directories) *before* dropping privs results in a fatal error as
slurmctld tries to create items which ultimately fail.  Moving
this process until after the privs and uid have changed allows
the process to succeed.

Reported by Jon Nelson <jdnelson@dyn.com>

Bug 2728

make it to the slurmctld when using message aggregation.

make sure we handle it correctly when the database comes back up.

    for better robustness.  This cray/select plugin code has
    been modified to remove a possible timing window where two
    aeld pthreads could exist, interfering with each other through
    the global aeld_running variable.

    An additional validity check has been added to the data provided
    to aeld through an alpsc_ev_set_application_info() call.
    If an error is returned from that call, only certain errors
    need the current socket connection closed to aeld and a new
    connection established.  Other error returns will log an
    error message and keep the current session established with
    aeld.

This might possibly be related to bug 2334, but it's a long shot.

slurm.conf instead of all.  If looking for specific addresses use
TopologyParam options No*InAddrAny.

This was broken in 15.08 with the advent of the referenced TopologyParams
the commits 9378f195 and c5312f52 are no longer needed.

Bug 2696

at the same time.

Bug 2683

Turns out making a variable static in a function will make it not safe
when dealing with threads.

With slurm-15.08.10, we're seeing occasional segfaults in slurmstepd. The logs point to the following line: slurm-15.08.10/src/slurmd/slurmstepd/mgr.c:2612

On that line, _get_primary_group() is accessing the results of getpwnam_r():
    *gid = pwd0->pw_gid;

If getpwnam_r() cannot find a matching password record, it will set the result (pwd0) to NULL, but still return 0. When the pointer is accessed, it will cause a segfault.

Checking the result variable (pwd0) to determine success should fix the issue.