If SLURM_BITSTR_LEN is set to some unreasonably large value it could
  result in the slurmctld allocating a huge memory buffer and consuming
  vast amounts of time.
Coverity CID 45264

CID 174162.

Coverity CID 45173

If one or more clusters in a federation are down, then print an
  appropriate warning and exit rather than cause the test to fail.
bug 4033

Mistake made in f9687bbc; strlcpy returns a count of chars copied
instead of a pointer to the string.

Otherwise the log will be spammed with "Buffer size limit exceeded".

Bug 3624.

Gets rid of one weird strlcpy call.

Ensure proper termination in places that were otherwise missing
it, and remove some awkward termination handling in other locations.

Starting from glibc-2.25 the macros major and minor are only available
from sys/sysmacros.h. This patch uses an autoconf macro to detect the
location and includes the header accordingly.

Bug 3982.

Unlike MPI, UCX requires each sender to have a unique tag, otherwise
messages will get mixed up.

Signed-off-by: Artem Polyakov <artpol84@gmail.com>

Fix direct connection table size to be equal to number of nodes
int the job instead of job step. This was triggering assert when,
for example, allocation was 16 nodes but srun was only using 2 of
them and not the first ones.

Signed-off-by: Artem Polyakov <artpol84@gmail.com>

middle of the other id_*.  I think it is ok to have the _id at the end for this
since we want the offset near it.

double ',' as we do with QOS in the TRES limits.  So I removed the logic to
check for it.

Bug 4019

Change to slurm_mutex_init
Coverity CID 171460

just unfreed memory, no real concern.

Replase "bool" with "int" as the return type of `pmixp_io_fd()`. This
was causing an interesting hidden bug affecting the performance.
Since this function was returning boolean value it was always
returning "1" instead of actual fd number. fd=1 is set to /dev/null
for slurmstepd, this is a char device that is always read/write ready
from the poll() perspective. So poll was continuously interrupting and
progress was fine, but CPU usage was ~100%.

Signed-off-by: Artem Polyakov <artpol84@gmail.com>

Signed-off-by: Artem Polyakov <artpol84@gmail.com>

(introduced in prev commit "Fix collective error path (timeout)"

Signed-off-by: Artem Polyakov <artpol84@gmail.com>

Signed-off-by: Artem Polyakov <artpol84@gmail.com>

UCP worker has to be re-armed after ucp_progress was called
on it. Original implementaition was done with wrong considerations
in mind.

Signed-off-by: Artem Polyakov <artpol84@gmail.com>

Signed-off-by: Artem Polyakov <artpol84@gmail.com>

Memory hooks are installed by UCX during it's load time. To prevent
that we need to `export UCX_MEM_MALLOC_HOOKS=no`.
With the previous approach ucx lib was loaded during pmix plugin
dlopen and we had no control over environment variables from the
plugin itself. The only working variant was to add mentioned variable
to slurmd's environment.
To improve user experience with this feature we want to be able to
transparently disable memory hooks from the plugin itself.

Signed-off-by: Artem Polyakov <artpol84@gmail.com>

Signed-off-by: Artem Polyakov <artpol84@gmail.com>

Signed-off-by: Artem Polyakov <artpol84@gmail.com>

Put request that is completed already directly to the _rcv_complete
list.
Ping-pong latency effect:
size    UCXv8   UCXv9
1       17.7    17.8
2       17.7    17.7
4       17.9    17.7
8       18.3    18.1
16      18.4    18.3
32      18.3    18.3
64      19.0    18.8
128     19.4    19.3
256     19.8    19.5
512     19.9    19.7
1024    19.7    19.6
2048    20.9    20.7
4096    23.7    23.5
8192    27.2    27.2
16384   30.3    30.0
32768   35.5    35.3
65536   47.4    47.2
131072  71.4    71.2
262144  2197.0  2202.2
524288  2727.1  2745.2
1048576 3580.4  3577.7
2097152 6120.7  6125.0
4194304 10329.2 10265.2

Signed-off-by: Artem Polyakov <artpol84@gmail.com>