First issue was identified on multi partition requests. job_limits_check()
was overriding the original memory requests, so the next partition
Slurm validating limits against was not using the original values. The
solution consists in adding three members to job_details struct to
preserve the original requests. This issue is reported in bug 4895.

Second issue was memory enforcement behavior being different depending on
job the request issued against a reservation or not.

Third issue had to do with the automatic adjustments Slurm did underneath
when the memory request exceeded the limit. These adjustments included
increasing pn_min_cpus (even incorrectly beyond the number of cpus
available on the nodes) or different tricks increasing cpus_per_task and
decreasing mem_per_cpu.

Fourth issue was identified when requesting the special case of 0 memory,
which was handled inside the select plugin after the partition validations
and thus that could be used to incorrectly bypass the limits.

Issues 2-4 were identified in bug 4976.

Patch also includes an entire refactor on how and when job memory is
is both set to default values (if not requested initially) and how and
when limits are validated.

Co-authored-by: Dominik Bartkiewicz <bart@schedmd.com>

If running without AccountingStorageEnforce but with the DBD and
it isn't up when starting the slurmctld you could get into a
corner case where you don't have a QOS list in the assoc_mgr.  Thus no
usage to transfer.

Bug 5156

Update slurm.spec and slurm.spec-legacy as well

Clang warns about a possible null dereference of job_part_ptr
if the !job_ptr->priority_array part of the conditional is taken.

Remove that part of the conditional, as it doesn't matter if that is
set or not here. The jobs eligibility on one vs. multiple partition is
not determined by that, but by the status of part_ptr_list and part_ptr.

Bug 5136.

Try to fill up each socket completely before moving into additional
sockets. This will minimize the number of sockets needed, improving
packing especially alongside MaxCPUsPerNode.

Bug 4995.

My mistake on commit 602817c8.

Bug 4922.

Without this, gang scheduling would incorrectly kick in for
these jobs since active_resmap has not been updated appropriately.

Bug 4922.

Code for this was removed in 2012.

Bug 5126.

Bug 5026.

Otherwise this will return the error message back to the next job submitter.

Bug 5106.

Bug 5106.

Link to CRIU as well.

Bug 4293.

Related to fix from bug 4155.

Bug 4155.

job_ptr->part_ptr is NULL if the partition has been deleted.

Crash only happens with PriorityFlags=CALCULATE_RUNNING enabled.

Bug 5136.

Bug 5146

Caused by a corrupted protocol_version field value being received
by the slurmstepd, as we cannot safely write/read a uint16_t across
the pipe as if it was an int.

Regression caused by commit 90b116c2.

Bug 5133.

Requeued jobs are marked as PENDING|COMPLETING until the epilog checks
in. The issue is that if job_set_alloc_tres gets called while in the
PENDING|COMPLETING state, the job's alloc_tres_str will be free'd. If
this job then gets checkpointed in this state (PENDING|COMPLETING + no
tres_alloc_str) on startup the controller would crash because it
expected the job to have a tres_alloc_str/cnt when in the COMPLETING
state. This could be triggered if starting the controller without the
dbd up. When the dbd comes up, the assoc_cache_mgr calls
_update_job_tres() which calls job_set_alloc_tres. It could also be
triggered by adding new tres.

This most likely started happening in 17.11.5 because of commit
865b672f which introduced calling _update_job_tres() on each job
after the dbd comes up.

Bugs 5137,4522

Only when the connection has timedout. If the connection is timing out,
consider increasing TCPTimeout in the slurm.conf

Bug 4574

Bug 5129.

Bug 5110.

Bug 5110.

Continuation of d0deea4f.

Bug 4841.

Use setenv() instead of setenvfs(), since setenvfs() memory allocation
is implemented with xmalloc() and fini_setproctitle() (which is called
on reconfigure) free's the memory with free(), leading to a:
"free(): invalid size" malloc_printerr error.

Continuation of dce83a23.

Bug 5095.

Due to current design the job limits are checked before the allocation is
made when one specifies a generic gres and a specific gres type is configured.
The workaround for now is to define a job submit plugin to control the user
request and succesfully apply limits.

Bug 4767

Bug 4960.

Bug 4887.

Can lead to deadlock within malloc depending on the exact timing.

Rework thread startup and shutdown code so pthread_cancel is not
needed.

Bug 5119, 5103.

happens.

Bug 5108

This reverts commit de5a4da2.

happens.

Bug 5108

Turns out the partititon's billing tres was working off the sum of
the node_ptrs which contain the max of all partitions they are in.

This isn't correct since each partition's billing can be different.

Set it correctly here.

No functional change.

The use in _watch_tasks needs to be removed as the switch to pthread_signal
from pthread_cancel means this will not get interrupted and would keep the
step alive for at least a second, potentially harming throughput.
Since the call to _poll_data() happens after the first timer expires,
this delay turns out to be unnecessary, so we won't be replacing it with
a pthread_cond_timedwait() construct.

The use jobacct_gather_stat_task() is unnecessary since the two locations
this can happen take place after _fork_all_tasks() has setup the tasks,
thus the delay should not be necessary.

Bug 5103.

These functions are not async-cancel-safe, and cannot safely be cancelled.
This leads to potential deadlock, either between our own locks, or deep
inside glibc when the thread held a malloc arena lock when canceled.

Replace with pthread_signal to the appropriate cond to
wake threads up at the appropriate time instead.

Bug 5103.