Patch by Bill Brophy, Bull.

You'll find this as commit 2079c03f58 to the chaos/slurm 2.3.3.chaos branch on March 7.  This is against SchedMD's master branch:

Patch by Don Lipari, LLNL.
https://github.com/chaos/slurm/commit/4de11bf0a8cd18207a60e7d3e1fa7a6fde0da431

Move a log message before fork() call in the new spank prolog/epilog
logic. The original code could deadlock in glibc when getting the time
to include in the message.

If the select plugin does not allocate memory to a job, then do not
try to alloocate memory to the job step, without this the step allocation
will always fail

Merge branch 'spank-epilog-for-schedmd' of https://github.com/grondo/slurm into grondo-spank-epilog-for-schedmd

Conflicts:
	NEWS

Fix bug in allocating GRES that are associated with specific CPUs. In some
cases the code allocated first available GRES to job instead of allocating
GRES accessible to the specific CPUs allocated to the job.

Mistakenly changed jobacct_gather_g_getinfo() into
jobacct_gather_g_setinfo()

This avoids conflicts with the "info" function.

This fixes a race condition in error handling logic added a couple
of days ago for slurmd/slurmstepd communications in commit
https://github.com/SchedMD/slurm/commit/ed31e6c7fdb5bcc1b0f0a8e3cbf5327604e64887

Access to secured FS often requires to have a valid token in the
user context. With SLURM, this token can be obtained using one of
the possible pluggable architecture, SPANK or PAM.

IO setup of SLURM can require to access secured FS (stdout/stderr
files). This patch ensures that pluggable frameworks are activated
and called prior to IO setup and that IO are terminated before
calling pluggable framework exit calls.

set PR_DUMPABLE as soon as possible, especially before any plugins
are loaded. This will allow someone debugging to get a coredump.

To prepare io_setup integration in _fork_all_tasks, error handling
must be transformed to not always return SLURM_ERROR but be prepared
to return SLURM_SUCCESS in case of an io_setup error.

Update spank(8) man page with documentation of new functionality,
including new callbacks: slurm_spank_slurmd_init, slurm_spank_slurmd_exit,
slurm_spank_job_prolog, and slurm_spank_job_epilog, as well as the new
spank_option_getopt() call for use in option processing by plugins.

Update spank header comments and documentation.

Add #defines for new slurmd and job prolog/epilog contexts, so that
their existence can be tested at compile time.

Add a new call to process spank options from a plugin.

The spank_option_getopt() function will search the current
spank environment for use of the option passed as an argument.
The current option cache, and the local environment are checked
for the use of the given spank option. This call is an alternative
to use of a global variable in combination with the option callback,
and is also needed for processing options in the isolated contexts
of slurm_spank_job_prolog() and slurm_spank_job_epilog().

Add spank_clear_remote_options_env() to clear any spank options
passed through the environment after they are no longer needed.
This is done in slurmd after running the spank job prolog || epilog,
as well as in the spank_post_opt function, after the env has been
searched for spank variables.

Always set spank options in the environemnt and spank job environment
to ensure that used options are propagated to the job prolog and
epilog.  (Previously, spank options were set in the environment
only in allocator context)

In slurmd and job prolog/epilog contexts, avoid loading plugins that
have no callbacks in the context in which they are loaded. That is
for slurmd, if there are no slurm_spank_slurmd_init or
slurm_spank_slurmd_exit callbacks, there is no reason to keep the
current plugin loaded.