e3a7bdcc
f9804256
d72b13f2

Reference bug 3366

If you are running on a Bluegene system we rely on the prolog to take us out of configuring
state.  These commits work good for system rebooting the nodes where the prolog is running,
but in the case of Bluegene this is the opposite desire :).   These commits on a Bluegene
pretty much make it so a batch job never gets launched.

Clear job's reason of "BeginTime" in a more timely fashion and/or prevents
    them from being stuck in a PENDING state. There are multiple ways of
    clearing the reason, especially on a lightly loaded system, but the
    state can persist indefinitely on a heavily loaded system.
bug 3368

Fix to logic for getting expected start time of existing job ID with
    explicit begin time that is in the past. Previous logic would
    compare that (past) begin time with advanced reservations that
    would compete with it rather than the current time.

On cray systems with step NHC, the step launches are delayed and
  produce a pair of messages (below) that caused the test to fail:
  srun: Job step creation temporarily disabled, retrying
  srun: Job step created

Avoid a test failing of all nodes in a partition are not usable
  (down, drained, reserved, or otherwise unusable).

Disable test if underlying select/linear use

Turns out this never worked, ever.  What used to happen is if the protocol_version that was
read in didn't match the rpc_version given to unpack things was just 0.  What this does
now is set the rpc_version to what was stored making it all good.

correctly.

Bug 3431

bug 3433

burst_buffer/cray - Fix race condition that could cause multiple batch job
    launch requests resulting in downed nodes.
bug 3366

This reverts commit b9bff82f.

It was leaking memory otherwise

Commit 63b7e3a8 changed the --mem limit to 1MB for the job if
not using a memory SelectType, but this can cause the job to fail
if the JobAcctGatherFrequency is frequent enough to notice that the
"sleep" command is using more than 1MB of resources.

Refactor test to avoid specifying job memory. Use --wrap to avoid
creating a temporary job script as well.

Commit 63b7e3a8 changed the --mem limit to 1MB for the step if
not using a memory SelectType, but this can cause the job to fail
if the JobAcctGatherFrequency is frequent enough to notice that the
"sleep" command is using more than 1MB of resources.

Refactor test to avoid specifying job memory. Use --wrap to avoid
creating a temporary job script.

Commit 63b7e3a8 changed the --mem limit to 1MB for the step if
not using a memory SelectType, but this can cause the job to fail
if the JobAcctGatherFrequency is frequent enough to notice that the
"sleep" command is using more than 1MB of resources.

Refactor test to avoid specifying memory memory; and since only one
step is checked for, only run a single step in the job. Use --wrap
to avoid creating a temporary job script.

Some portions of tests 21.30 and 21.34 failed with accounting and
priority basic. These changes disable portions of those tests as
needed based upon configuration.

test1.63 was failing periodically due to a race condition. A signal
  was being sent to srun before the signal handler thread was spawned.

Reset a job's memory limit based upon what's available after node
  reboot, which can change on a KNL if the MCDRAM mode is changes
  on reboot

This bug was likely the root cause of bug 3366. If the backfill scheduler
  allocates resources for a batch job and a node reboot is required, the
  batch launch RPC would be sent to the agent. At that point, there is a
  race condition between the agent and the job_time_limit() function
  testing for boot completion. If the job_time_limit() function ran
  first, it would trigger a second launch RPC request getting sent to
  the agent.
bug 3366

Clean up logic to test if job is configuring
bug 3366

Do not launch a batch step while the job is configuring. Previous
  logic checked for the PrologSlurmctld running, but not nodes
  booting. Checking the job's CONFIGURING state flag will validate
  both.
bug 3366

Add check to avoid step allocation logic from executing job
  configuration completion logic multiple times (check if job
  is configurating before clearing flag and resetting time limit).
bug 3366

slurmctld/agent race condition fix: Prevent job launch while PrologSlurmctld
    daemon is running or node boot in progress.
bug 3366

This is required to manage the configuration completion.
bug 3366

This will be required to lock the job structure
bug 3366

Remove the return value from the agent_retry() function. It is not
  used anywhere and needs to be removed to run as a pthread.
bug 3366