With release_agent notified at the step cgroup level, the step cgroup
can be removed while slurmstepd as not yet finished its internals
epilog mechanisms. Inhibiting release agent at the step level and
ensuring its proper removal helps to guarantee that the node will only
be eligible for job execution when the resources will be completely
available (no longer used by the job or the epilogs).

A delay occurs between a task creation and its addition to a different
cgroup than the inherited one. In the meantime, the process can disapear
resulting in a ESRCH during the addition in the second cgroup. Now react
to that event as a warning instead of an error.

Move the code that waits for parent signal before exec(2) out of
exec_task() and into fork_all_tasks() directly. This makes all
the code that handles the fork-and-wait into slurmstepd/mgr.c,
and allows the exec_wait_child_wait_for_parent() function to
be used in place of explicit read().

tty setup needs to occur before child tasks block waiting from signal
to the parent, so move this code out of exec_task() into fork_all_tasks()
so that the wait-for-signal-from-parent code can also later move out
of exec_task().

As reported by Sam Lang on slurm-dev, task_epilog scripts are not
held before exec, and thus there is a race condition between when
the task_epilog is launched and slurmstepd calls slurm_container_add()
during which the task_epilog script could either run to completion, or
launch other processes that escape any job container defined by
configuration.

Use the new "exec_wait" api to have the child wait before exec just
as is done in fork_all_tasks.

Based on an original idea by Sam Lang <samlang@gmail.com>.

Remove the explicitly coded fork-and-wait-before-exec code from
slurmstepd fork_all_tasks and replace with the "exec_wait" API.
This change should be functionally identical to the previous
code.

Abstract the code in slurmstepd fork_all_tasks that allows the parent
to signal children before they call exec into an "exec_wait_info"
interface. This will allow the code to be easily reused in other
parts of slurmstepd (e.g. task epilog) without cut-and-paste of code.

Prevent job hold by operator or account coordinator of his own job from
being an Administrator Hold rather than User Hold by default.

Prevent slurmctld crashing with divide by zero with a configuration of MaxMemPerCPU=0.

block happens correctly now.

Major re-write of the CPU Management User and Administrator Guide (web
page) by Martin Perry, Bull.

PluginDir is a path. It shouldn't be an error to have duplicate
plugins in your path. Plus, the error is not helpful because it
doesn't specify which path is not being loaded. Therefore, just
remove the error and load the first plugin in the path as expected.

Fix bugs in sched/backfill with respect to QOS reservation support and job
time limits. Patch from Alejandro Lucero Palau (Barcelona Supercomputer Center).

Fix to GRES allocation logic when resources are associated with specific
CPUs on a node. Patch from Steve Trofinoff, CSCS.

(i.e. 1-9,0 instead of 0-9).  The bug would cause 'sacct -N nodename' to
not give correct results on these systems.

is in an error state, won't deny jobs.

restarts of the slurmctld.

admin sets the state to error.

Advise use of the logrotate tool in order to avoid SLURM log files
from growing too large. Patch from Rod Shultz, Bull.

Do not treat the absence of a gres.conf file as a fatal error on systems
configured with GRES, but set GRES counts to zero. These counts can be
Counts can be altered by node_config_load() in the gres plugin.

The slurmctld code that processes job notify messages unecessarily
restricts these messages to be from the slurm user or root. This
patch allows users to send notifications to their own jobs.

Many cosmetic modifications to eliminate warning message from GCC version
4.6 compiler, mostly due to unused variables.

jobs happen to be running on blocks not in the new config.

salloc/mpirun does not play well together with task affinity socket binding.  The following example illustrates the problem.

[sulu] (slurm) mnp> salloc -p bones-only -N1-1 -n3 --cpu_bind=socket mpirun cat /proc/self/status | grep Cpus_allowed_list
salloc: Granted job allocation 387
--------------------------------------------------------------------------
An invalid physical processor id was returned ...

The problem is that with mpirun jobs Slurm launches only a single task, regardless of the value of -n. This confuses the socket binding logic in task affinity.  The result is that task affinity binds the task to only a single cpu, instead of all the allocated cpus on the socket.  When mpi attempts to bind to any of the other allocated cpus on the socket, it gets the "invalid physical processor id" error. Note that the problem may occur even if socket binding is not explicitly requested by the user.  If task/affinity is configured and the allocated CPUs are a whole number of sockets, Slurm will use "implicit auto binding" to sockets, triggering the problem.
Patch from Martin Perry (Bull).

Update reservation web page to describe mechanism to reserve CPUs rather than whole nodes and provide an example.

Avoid clearing a job's reason from JobHeldAdmin or JobHeldUser when it is
otherwise updated using scontrol or sview commands. Patch based upon work
by Phil Eckert (LLNL).

Do not remove the backup slurmctld's pid file when it assumes control, only
when it actually shuts down. Patch from Andriy Grytsenko (Massive Solutions
Limited).