the queue when trying to place a larger than midplane job.

In cray/srun wrapper, only include aprun "-q" option when srun "--quiet"
option is used.

Here's what seems to have happened:

- A job was pending, waiting for resources.
- slurm.conf was changed to remove some nodes, and a scontrol reconfigure was done.
- As a result of the reconfigure, the pending job became non-runnable, due to "Requested node configuration is not available". The scheduler set the job state to JOB_FAILED and called delete_job_details.
- scontrol reconfigure was done again.
- read_slurm_conf called _restore_job_dependencies.
- _restore_job_dependencies called build_feature_list for each job in the job list
- When build_feature_list tried to reference the now deleted job details for the failed job, it got a segmentation fault.

The problem was reported by a customer on Slurm 2.2.7.  I have not been able to reproduce it on 2.4.0-pre3, although the relevant code looks the same. There may be a timing window. The attached patch attempts to fix the problem by adding a check to _restore_job_dependencies.  If the job state is JOB_FAILED, the job is skipped.

Regards,
Martin

This is an alternative solutionh to bug316980fix.patch

Added default_time field in partition records in Lua job submit
plugin.

Added a new Lua library name to try loading with dlopen() in Lua
based plugins.

Only report "gres/<name> lacks File parameter" if some nodes define
File AND this node does not AND (new part here) the GRES count on
this node is non-zero

Fixes for jobs with long argument lists

Replace two xstrcat() calls per argument with a single xmalloc()
call. This significantly speeds up handling of REQUEST_JOB_INFO RPCs
when some jobs have long argument lists.

Change argc from uint16_t to uint32_t in slurmctld and
slurmstepd. Rest of the code already use uint32_t for argc.

Fixes a bunch of warnings of this type
warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body

In current version of slurm initscript, a stop action returns a
non null exit code as slurmstatus exit code is directly used and
the daemons are stopped.
Ensure that when called from slurmstop, slurmstatus error code is
reversed to correctly match the attended error code of the stop
stage.
Port of v2.4 commit a09bffa5
Matthieu Hautreux authored 3 months ag

Patch from Aleksej Saushev.

c0a7a7a4

is a convenience function in BSD and glibc that internally calls
the equivalent of

    int masterfd = open("/dev/ptmx", flags);
    grantpt (masterfd);
    unlockpt (masterfd);
    int slavefd = open (slave, O_RDRW|O_NOCTTY);

(in psuedocode)

On Linux, with some combinations of glibc/kernel (in this
case glibc-2.14/Linux-3.1), the equivalent of grantpt(3) was failing
in slurmstepd with EPERM, because the allocated pty was getting
root ownership instead of the user running the slurm job.

From the POSIX description of grantpt:

 "The grantpt() function shall change the mode and ownership of the
  slave pseudo-terminal device... The user ID of the slave shall
  be set to the real UID of the calling process..."

 http://pubs.opengroup.org/onlinepubs/007904875/functions/grantpt.html

This means that for POSIX-compliance, the real user id of slurmstepd
must be the user executing the SLURM job at the time openpty(3) is
called. Unfortunately, the real user id of slurmstepd at this
point is still root, and only the effective uid is set to the user.

This patch is a work-around that uses the (non-portable) setresuid(2)
system call to reset the real and effective uids of the slurmstepd
process to the job user, but keep the saved uid of root. Then after
the openpty(3) call, the previous credentials are reestablished
using the same call.

Fix for srun allocating running within existing allocation with --exclude
option and --nnodes count small enough to remove more nodes.

    > salloc -N 8
    salloc: Granted job allocation 1000008
    > srun -N 2 -n 2 --exclude=tux3 hostname
    srun: error: Unable to create job step: Requested node configuration is not available

Patch from Phil Eckert, LLNL.

Fix bug in step task distribution when nodes are not configured in numeric
order. Patch from Hongjia Cao, NUDT.

Fix bug when requeued batch job is scheduled to run on a different node
zero, but attemts job launch on old node zero causing fatal error
"Invalid host_index -1 for job #"

Avoid slurmctld abort due to bad pointer when setting an advanced
reservation MAINT flag if it contains no nodes (only licenses).

blocks are in an error state.