The test21.30/inc21.30.11 does not work if SelectTypeParameters
does not manage memory (not a commonly used configuration), but
this adds a check to prevent a failure that is due to the
configuration rather than slurm code bug.

These tests confirms correct CUDA percentages in an environment
where the gres/mps counts are not always 100 and can very from
device to device within a node.

Slurmd was generating errors of this sort for all GRES:
error: Ignoring gres.conf record, invalid name: mps
This was due to some GRES setup logic not being invoked
on the slurmd. In any case, the relevant checking now only
happens on the slurmctld.

This patch fixes the regression caused by 07483564 where only the
linux plugin was handling this scenario correctly.

Bug 6332

If a gres type was identified in the node's GRES field, but that
type was not listed in the slurm.conf GresTypes parameter then
those GRES would appear in the output of "scontrol show node".
This misleading/wrong information would cause a bunch of tests
to fail. This patch only moves valid node GRES into the node
record and deletes the rest.

This logic supports different MPS counts on each device, which
is possible on a heterogeneous system

This prints GRES configuration information and exits. For testing.

Needed for testing of gres/mps allocations

Fix formatting and typo in log message

Wrong variable name used

This works for heterogeneous systems, various Counts for each GPU
type, steps which use a subset of the job's gres/mps allocation,
etc.

Bug 4700.

Co-authored-by: Tim Wickberg <tim@schedmd.com>

Bug 6311.

Silences this useless warning:

ar: `u' modifier ignored since `D' is the default (see `U')

Bug 6209.

Bug 5820.

Bug 6331

convert src/common/gres.c logic to use new suffix_mult()
function in src/common/slurm_protocol_defs.c to parse
numeric sufficies (GB, MB, etc.). No change in functionality.

possibly needed to update a partition's total TRES counts when
updating a node's GRES counts

Added "Ben Roberts" to the team list

Bug #6336

This patch removes the MemLimitEnforce parameter from Slurm and moves the
condition to enable the feature to the JobAcctGatherParam=OverMemoryKill.
Setting OverMemoryKill will now enable the mechanism that was before enabled
with MemLimitEnforce=yes, so activation of killing jobs or steps due to
exceeding memory will be done in the same place. Also reconfiguring or
starting slurmd/ctld with incorrect configuration will cause a fatal error.

Bug 5479

Redundant logic committed in 62e378a4
bug 6337

this fixes bug introduced in commit 4facada1 when a job was
submitted with a valid gres name but count of zero

We decided with AMAT to remove gres/mps percent specification.

Add a debug statement.

Bug 5823

When the backup slurmdbd is active, there is no need to print
connection errors every 10 minutes when the backup checks in on the
primary, at least at the 'error' level.

Prevent dbd clients, like sacctmgr, from printing a connection error
when running commands against the backup slurmdbd unless debug2.
Before, the clients would always print errors when trying to access
the primary dbd first.

Bug 5823

The GRES Links info was added to v18.08, but the unpack was
missing that field in v19.05.
bug 6337

Issuing job cancel on the wrong job ID. Also increase sleep in a
test to better try to overlap the jobs.

Support GRES types that include numbers (e.g. "--gres=gpu:123g:2").
Also change number suffix processing to use common function
See commit 396e82a167
bug 6345

See commit 396e82a167
bug 6345