On switch resource allocation failure, free partial allocation.
Failure mode was CAU could be allocated on some nodes, but not
others. The CAU allocated on nodes and switches up to the failure
point were never released.

Fix logic bugs for SchedulerParameters option of max_rpc_cnt.
Scheduling would be delayed for job arrays and
backfill scheduling would be disabled unless max_rpc_cnt > 0.

on Mixed state.

This largely reverts commit 0ec2af27
just to cut down on some logging

in it.

Signed-off-by: Danny Auble <da@schedmd.com>

Rather than treat invalid SchedulerParameters options as a fatal
error, print an error and use to the default value.

Disables job scheduling when there are too many pending RPCs

This is related to defering batch job scheduling if there
are a bunch of requests pending

rather than 1 sec interval retries

If pthread_create call fails, decrease sleep before retry from
1 sec to 0.1 sec

This also reverts commit 8cff3b08 and
ced2fa3f

slurmdbd plugin.

9368ff2d

new associations were added since it was started.

Permit multiple batch job submissions to be made for each run of the
scheduler logic if the job submissions occur at the nearly same time.
bug 616

Decrease maximimum scheduler main loop run time from 10 secs to
4 secs for improved performance.
If running with sched/backfill, do not run through all jobs on
periodic scheduling loop, but only the default depth. The
backfill scheduler can go through more jobs anyway due to its
ability to relinquish and recover locks.
See bug 616

if an job step's network value is set by poe, either by directly
executing poe or srun launching poe, that value was not being
propagated to the job step creation RPC and the network was not
being set up for the proper protocol (e.g. mpi, lapi, pami, etc.).
The previous logic would only work if the srun execute line
explicitly set the protocol using the --network option.

Do not overcommit partitions with PreemptMode=off

Prevent preemption of jobs in partition where PreemptMode=off

processes.

When slurmctld restarted, it would not recover dependencies on
job array elements and would just discard the depenency. This
corrects the parsing problem to recover the dependency. The old code
would print a mesage like this and discard it:
slurmctld: error: Invalid dependencies discarded for job 51: afterany:47_*

be setup for 1 node jobs.  Here are some of the reasons from IBM...

1. PE expects it.
2. For failover, if there was some challenge or difficulty with the
   shared-memory method of data transfer, the protocol stack might
   want to go through the adapter instead.
3. For flexibility, the protocol stack might want to be able to transfer
   data using some variable combination of shared memory and adapter-based
   communication, and
4. Possibly most important, for overall performance, it might be that
   bandwidth or efficiency (BW per CPU cycles) might be better using the
   adapter resources.  (An obvious case is for large messages, it might
   require a lot fewer CPU cycles to program the DMA engines on the
   adapter to move data between tasks, rather than depend on the CPU
   to move the data with loads and stores, or page re-mapping -- and
   a DMA engine might actually move the data more quickly, if it's well
   integrated with the memory system, as it is in the P775 case.)

than you really have.

doesn't get chopped off.