This bug was observed once running test1.103 and found to be
reproducible under some circumstances. How this works is when
a job is submitted with a --time-min value, then the backfill
scheduler set's the job's time limit to that value (see below),
tries to start it, and if successful then set the time limit to
the largest value possible without delaying the expected start
time of any higher priority jobs. If the job can't be started,
the job's (maximum) time limit is supposed to be restored to
its previous value. That was happening in some, but not all
places in the code. This patch rests the time limit in the missing
cases.

Note: The job's time limit is set to the --time-min value here:
diff --git a/src/plugins/sched/backfill/backfill.c b/src/plugins/sched/backfill/backfill.c
index 5600495e1d..e22a810394 100644
--- a/src/plugins/sched/backfill/backfill.c
+++ b/src/plugins/sched/backfill/backfill.c
@@ -1930,7 +1930,7 @@ next_task:
 		    slurm_get_preempt_mode())
 			time_limit = job_ptr->time_limit = 1;
 		else if (job_ptr->time_min && (job_ptr->time_min < time_limit))
-			time_limit = job_ptr->time_limit = job_ptr->time_min;
+			time_limit = job_ptr->time_limit = job_ptr->time_min;		// SET HERE

 		later_start = now;

Bug 6787.

What can happen without this sleep is the "JOBID" output from
task 0 comes after the "HOST" output from task 1, which breaks
the parsing just below this.

Make sure that a job using gres/mps and gres/gpu on the same
node either run at different times or use different GPUs.
This restores part of test40.2 that was bad and removed in
commit 5a6409bb. It also insures that the logic added in
commit ac1182cf works as desired.

Without this change if core binding was not specified (i.e. no
cores/cpus in the gres/conf file for gpu), then gres/mps could
be allocated to overlap with gres/gpu that were already allocated.

Bug 4188.

Bug 6608.

Bug 6807.

==8640== Thread 5 bckfl:
==8640== Syscall param openat(filename) points to unaddressable byte(s)
==8640==    at 0x4A81D0E: open (open64.c:48)
==8640==    by 0x5934ABB: _update_job_env (burst_buffer_cray.c:3338)
==8640==    by 0x5934ABB: bb_p_job_begin (burst_buffer_cray.c:3962)
...
==8640==  Address 0x6b96120 is 16 bytes inside a block of size 61 free'd
==8640==    at 0x48369AB: free (vg_replace_malloc.c:530)
==8640==    by 0x49D4873: slurm_xfree (xmalloc.c:244)
==8640==    by 0x490C317: free_command_argv (run_command.c:249)
==8640==    by 0x5934A5C: bb_p_job_begin (burst_buffer_cray.c:3947)
...
==8640==  Block was alloc'd at
==8640==    at 0x4837B65: calloc (vg_replace_malloc.c:752)
==8640==    by 0x49D4566: slurm_xmalloc (xmalloc.c:87)
==8640==    by 0x49D4B67: makespace (xstring.c:103)
==8640==    by 0x49D4C91: _xstrcat (xstring.c:134)
==8640==    by 0x49D4ECF: _xstrfmtcat (xstring.c:280)
==8640==    by 0x593497C: bb_p_job_begin (burst_buffer_cray.c:3936)
...

Bug 6807.

Bug 6807.

Bug 6807.

Bug 6807.

Changed the behavior of "scontrol reboot" to require the user to specify
the nodes to reboot rather than defaulting to ALL.

Bug 6465

This corrects the gres/mps test to insure that CUDA_VISIBLE_DEVICES
is always zero (it is dependent upon the devices under MPS control
and not related to cgroup constrained devices). Also correct some
logic related to how the percentage calculation works in the test.

Permit any GPU to be used for gres/mps mode, but only
one GPU can be used

a request for --gres=mps:1 (specifically with a count of one)
was in some places being treated like a request for a full GPU

The variable is relative to which GPUs are managed by
MPS. Currently Slurm only allows one GPU to be managed
by MPS at a time, so the env var should always be zero.

Allow it to support use of any GPU in multi-GPU system

This allows jobs to be placed on booting nodes rather than being given a
whole node even if it would have been better to wait for the node boot.

Bug 6782

to make nodes available after being suspended even if down, drain,
failed.

Bug 6212

Bug 6333

Bug 6333

Rely on POWERING_DOWN bit. This allows POWERING_DOWN nodes to be cleared
after a restart -- since suspend_node_bitmap was local to power_save.c.

Bug 6333

Bug 6333

Instead of just guessing the time, let's use the original time.

Bug 6333

While "powering down" the nodes aren't eligible to be allocated.
Nodes will remain "powering down" for SuspendTimeout time.

Bug 6333

NODE_STATE_POWER_SAVE == node is actually off

Bug 6333

Run suspend and resume more often than ResumeTimeout after last suspend.
Don't allocate suspending nodes until after SuspendTimeout.

Bug 6333

If the MPS server is started with the environment variable
CUDA_MPS_ACTIVE_THREAD_PERCENTAGE, then the MPS server will
be limited to the percentage of the GPU total, which will not
work as desired if additional jobs are initiated.

User percentage logic was incorrect and unnecesarry.
Removed the logic and associated test.

Bug 5667