If GRES configuration data is unavailable from gres.conf, then use the
    node's "Gres=" information slurm.conf. This will eliminate or minimize the
    gres.conf file in many situations.
bug 6761

They were a bit too verbose for my taste

In 0e149092 not setting the variable when job was not requesting any gres
was considered a bug. The cuda API will use all devices if the variable is
not set. If it is set to some unknown or empty value, it will use no devices.
This variable should be used only for testing purposes and ConstrainDevices=yes
in cgroup is recommended.

Bug 6412

gres plugins will setup environment for every gres in the system even if the
job has not requested it.

Bug 6412

than one GRES of the same name but different type"

This reverts f7fca7ba

Bug 6412

initial work needed for bug 6761 support

comment format and change some log messages

this problem was triggered with a configuation of
PrologFlags=Alloc,Contain

This eliminates a slurmctld error message when a job shrinks to
size zero. There is no need to re-compute the CPU count and the
job_resources node_bitmap is empty. Logic works fine without
this change if job size shrinks, but not to size zero.
bug 6472

When a job size was reset to zero, this error message was printed:
slurm_allocation_lookup: Job/step already completing or completed
which may lead the user to believe the operation failed when it
worked as planned. Now it prints this:
To reset Slurm environment variables, execute
  For bash or sh shells:  . ./slurm_job_43565_resize.sh
  For csh shells:         source ./slurm_job_43565_resize.csh
Where the reset scripts contain zero node count information:
export SLURM_NODELIST=""
export SLURM_JOB_NODELIST=""
export SLURM_NNODES=0
export SLURM_JOB_NUM_NODES=0
export SLURM_JOB_CPUS_PER_NODE=""
unset SLURM_NPROCS
unset SLURM_NTASKS
unset SLURM_TASKS_PER_NODE

Continuation of 2764f3fd

Bug 6589

Continuation of 9a243a1a

Bug 6592

Many comments were modified to follow Linux kernel standard
Many log messages were using the old function name and now
  print __func__ instead
A few log messages lacked the function name and those were added

No change in any logic

This adds logic to validate the count of GRES by device Type and
not just Name and modifies the data structures as needed for
consistency within slurmctld.

Removed linear search, replaced with direct record references and a hashmap.
This is faster and avoids potential collisions between assoc id's and user id's.
Bug 4811

Fixed existing usages as well.
Bug 4811

Coverity CID 197448
bug 6303

Remove reference to REQUEST_SIGNAL_PROCESS_GROUP in slurmstepd.
It has been defunct since July 2013

sort the expected and actual output for GRES APIs irrelevant.
Depending upon the GRES plugins loaded (specifically gres/gpu plus
gres/mps), the GRES records can be sorted by File name to insure
the GRES records line up (the same position in both lists should
refer to the same device file).

Bug 6750.

This logic could allocate to a job a GRES device with
an availability count of zero.

This should only happen if there is flawed logic somewhere, but
avoiding an abort is better than not.

If the count of GPUs configured in slurm.conf and gres.conf differ
and FastSchedule>=1 then the bitmap identifying the GPU allocation
sent from slurmctld to slurmd will differ. Previously this resulted
in CUDA_VISIBLE_DEVICES being set to NULL. Now it will be set correctly.

bug 6725

If slurmd finds GRES with files and slurmctld can't use them
(i.e. slurm.conf has a GRES count of 0), then avoid trying to
create zero length bitmaps in the GRES data structure.
bug 6725

This makes the gres bitmap size equal to the number of records for
shared gres (i.e. gres/mps), otherwise it is the gres count (i.e.
gres/gpu).

bug 6733

if the device files for gres/gpu are out of order or grouped
in an unordered fashion (e.g. "Name=gpu Files=/dev/nvidia[2,8,10]")
then split the gres/gpu records to one record per file and make
sure the gres/mps records are in an identical order. Required for
matching gres/gpu and gres/mps records (one GPU can be allocated
either as gres/gpu or as gres/mps, but not both, so we need to
be able to match records in slurmctld).

Coverity CID 197447

Bug 6710.

Bug 6590.

Make some tests better able to work with CR_ONE_TASK_PER_CORE