cnode counts.

Switch/nrt - Properly track usage of CAU and RDMA resources with multiple
tasks per compute node. Previous logic would allocate resources once per
task and then deallocate once per node, leaking CMA and RDMA resources
and preventing their use by future jobs.

On switch resource allocation failure, free partial allocation.
Failure mode was CAU could be allocated on some nodes, but not
others. The CAU allocated on nodes and switches up to the failure
point were never released.

Fix logic bugs for SchedulerParameters option of max_rpc_cnt.
Scheduling would be delayed for job arrays and
backfill scheduling would be disabled unless max_rpc_cnt > 0.

on Mixed state.

in it.

Signed-off-by: Danny Auble <da@schedmd.com>

Disables job scheduling when there are too many pending RPCs

This also reverts commit 8cff3b08 and
ced2fa3f

new associations were added since it was started.

Permit multiple batch job submissions to be made for each run of the
scheduler logic if the job submissions occur at the nearly same time.
bug 616

if an job step's network value is set by poe, either by directly
executing poe or srun launching poe, that value was not being
propagated to the job step creation RPC and the network was not
being set up for the proper protocol (e.g. mpi, lapi, pami, etc.).
The previous logic would only work if the srun execute line
explicitly set the protocol using the --network option.

Prevent preemption of jobs in partition where PreemptMode=off

processes.

When slurmctld restarted, it would not recover dependencies on
job array elements and would just discard the depenency. This
corrects the parsing problem to recover the dependency. The old code
would print a mesage like this and discard it:
slurmctld: error: Invalid dependencies discarded for job 51: afterany:47_*

be setup for 1 node jobs.  Here are some of the reasons from IBM...

1. PE expects it.
2. For failover, if there was some challenge or difficulty with the
   shared-memory method of data transfer, the protocol stack might
   want to go through the adapter instead.
3. For flexibility, the protocol stack might want to be able to transfer
   data using some variable combination of shared memory and adapter-based
   communication, and
4. Possibly most important, for overall performance, it might be that
   bandwidth or efficiency (BW per CPU cycles) might be better using the
   adapter resources.  (An obvious case is for large messages, it might
   require a lot fewer CPU cycles to program the DMA engines on the
   adapter to move data between tasks, rather than depend on the CPU
   to move the data with loads and stores, or page re-mapping -- and
   a DMA engine might actually move the data more quickly, if it's well
   integrated with the memory system, as it is in the P775 case.)

than you really have.

doesn't get chopped off.

    a minus sign for options was intended.

Some of these were resulting in the state of a job not being updated
correctly to tools like sview.

in waiting reason ReqNodeNotAvail.

Add support for job array options in the qsub command, in #PBS
options for sbatch scripts and set the appropriate environment
variables in the spank_pbs plugin (PBS_ARRAY_ID and PBS_ARRAY_INDEX).
Note that Torque uses the "-t" option and PBS Pro uses the "-J"
option.

slurm.conf.  Rebooting daemons after adding nodes to the slurm.conf
is highly recommended.

Perhaps should also look into doing this for nodeinfo and libstate

this would cause pmd's to hang.