Do not mark the node DOWN if its memory or tmp disk space is lower
than configured, just log it using debug message type

If a job is explicitly suspended, its priority is set to zero.
This resets the priority when requeued and also documents that
if the job is requeued (e.g. due to a node failure), then it
is placed in a held state.

Without this patch, the job's RunTime includes its RunTime from
before it's prior suspend (i.e. the job's full RunTime rather than
just the RunTime of the requeued job).

Hello,

I think I found another bug in the code (I'm using 2.6.3 but I checked the 2.6.5 and 14.03 versions and it's the same there).

In file sched/backfill/backfill.c:

1)
_add_reservation function, from lines 1172:

if (placed == true) {
        j = node_space[j].next;
        if (j && (end_reserve < node_space[j].end_time)) {
                /* insert end entry record */
                i = *node_space_recs;
                node_space[i].begin_time = end_reserve;
                node_space[i].end_time = node_space[j].end_time;
                node_space[j].end_time = end_reserve;
                node_space[i].avail_bitmap =
                        bit_copy(node_space[j].avail_bitmap);
                node_space[i].next = node_space[j].next;
                node_space[j].next = i;
                (*node_space_recs)++;
        }
        break;
}
I draw a picture with `node_space` state after 2 iterations (see attachment).

In case where the new reservation is fully inside another reservation,
then everything is OK.
But if the new reservation spans multiple existing reservations then the `end entry record` is not created.
This is because only the newly created `start entry record` is checked.

Easy fix would be to change the if into a loop, for example:

if (placed == true) {
    while((j = node_space[j].next) > 0) {
        if (end_reserve < node_space[j].end_time) {
           //same as above
           break;
        }
    }
    break;
}

2)
You could also change line 612:
        node_space = xmalloc(sizeof(node_space_map_t) *
                             (max_backfill_job_cnt + 3));
To `(max_backfill_job_cnt * 2 + 1)` , since each reservation can add at most two entries (check at line 982 should never execute). At the moment, in a worst case scenario this only checks half of the max_backfill_job_cnt.

NOTE: However this is all based on the assumption, that it is not done on purpose to speed up the calculations and trading some of the accuracy (especially point 2).

Best regards,
Filip Skalski

for better debug

midplane block that starts on a higher coordinate than it ends (i.e if a
block has midplanes [0010,0013] 0013 is the start even though it is
listed second in the hostlist).

It has been changed to improve the calculated value for pending
jobs and use the actual node count value for jobs that have been
started (including suspended, completed, etc.)
bug 549

being in error.

will return ENOTCONN and not initialize the addr_str causing valgrind
errors.

This allows multiple job ids to hold, uhold, resume, suspend, release, etc.

Fix slurmstepd race condition when separate threads are reading and
modifying the job's environment, which can result in the slurmstepd failing
with an invalid memory reference. Observed at shutdown when trying
to run the task epilog and trying to read the env var:
SLURM_STEP_KILLED_MSG_NODE_ID

Without this patch, free() is called on a random memory location
(i.e. whatever is on the stack), which can result in slurmstepd
dying and a completed job not being purged in a timely fashion.

Fix race condition in authentication credential creation that could corrupt
memory. (NOTE: This race condition has existed since 2003 and would be
exceedingly rare.)

This is needed for job arrays with discontiguous task ID values
(e.g. "123_[1,3,5,...99999]")

Previously job arrays were only listed with their native job ID
(e.g. 123_0 listed as 123, 123_1 as 124, etc). Now lists the job ID
using both format (e.g. "123_1 (124)"). The same format is used
for job step IDs (e.g. "123_1.2 (124.2)").

This adds a mechanism to kill a hung apbasil command

news.html.

Previous logic never reopened the file, preventing proper functioning
of logrotate.

Use hash function to locate job records for improved performance.

Change partition write lock to a read lock as we use a different
mechanism for hidden partitions in getting individual jobs.

Correct logic returning remaining job dependencies in job information
reported by scontrol and squeue. Eliminates vestigial descriptors with
no job ID values (e.g. "afterany"). As depdencies are removed, the
job ID values were removed from the strings, but not the descriptors.
This eliminates both. It also checks the full job ID to make sure we do
not remove "afterany:1234" when job "123" completes.

Fix race condition on batch job termination that could result in a job exit
code of 0xfffffffe if the slurmd on node zero registers its active jobs at
the same time that slurmstepd is recording the job's exit code.
but 535

proctrack/cgroup - Add locking to prevent race condition where one job step
is ending for a user or job at the same time another job stepsis starting
and the user or job container is deleted from under the starting job step.
bug 447

Substantial performance improvement for systems with Shared=YES or FORCE
and large numbers of running jobs (replace bubble sort with quick sort).
Bug 525

Original code worked only for Cray systems. For other systems it
set gres_alloc to the total number of each GRES allocated on each
node to any job

bug 511