Split a slurmctld's job record "shared" field into "share_res"
(share resource) and "whole_node" fields. Needed to better manage
allocation of whole nodes for core specialization without disabling
gang scheduling of such jobs.

to suspend/resume array elements.

This reverts commit 34fd501c.

connect in a loop instead of producing a fatal.

This reverts commit 2fa28eb6.

Conflicts:

	NEWS

data correctly accumulating differences between sampling intervals.
Fix the data structure mismatch between acct_gather_filesystem_lustre.c
and slurm_jobacct_gather.h which caused the hdf5 plugin to log incorrect
data.

New configuration parameter of CoreSpecPlugin.
Added core_spec/none and core_spec/cray plugins (just stubs for now).

the srun help.

not "idle" when in a reservation.

Add version number to node and front-end configuration information visible
using the scontrol tool. Sview and sinfo still need to be changed.

Add specialized core count field to job credential data.
NOTE: This changes the communications protocol from other pre-releases of
version 14.03. All programs must be cancelled and daemons upgraded from
previous pre-releases of version 14.03. Upgrades from version 2.6 or earlier
can take place without loss of jobs

network traffic accounting plugin.

to print more information when debug and when io error occur.

add/remove columns.

caused by commit 68f0f5db

Do not reset a job's priority when the slurmctld restarts if previously
set to some specific value.
bug 561

groups.

hostlist_push_host().

are not longer set DOWN, they are set to DRAIN instead.

Core specialization is now fully supported.

This reverts commit 3464295e.

Make sure that licenses are not oversubscribed in overlapping
reservations.

Do not mark the node DOWN if its memory or tmp disk space is lower
than configured, just log it using debug message type

parameter in slurm.conf.

If a job is explicitly suspended, its priority is set to zero.
This resets the priority when requeued and also documents that
if the job is requeued (e.g. due to a node failure), then it
is placed in a held state.

Without this patch, the job's RunTime includes its RunTime from
before it's prior suspend (i.e. the job's full RunTime rather than
just the RunTime of the requeued job).

Hello,

I think I found another bug in the code (I'm using 2.6.3 but I checked the 2.6.5 and 14.03 versions and it's the same there).

In file sched/backfill/backfill.c:

1)
_add_reservation function, from lines 1172:

if (placed == true) {
        j = node_space[j].next;
        if (j && (end_reserve < node_space[j].end_time)) {
                /* insert end entry record */
                i = *node_space_recs;
                node_space[i].begin_time = end_reserve;
                node_space[i].end_time = node_space[j].end_time;
                node_space[j].end_time = end_reserve;
                node_space[i].avail_bitmap =
                        bit_copy(node_space[j].avail_bitmap);
                node_space[i].next = node_space[j].next;
                node_space[j].next = i;
                (*node_space_recs)++;
        }
        break;
}
I draw a picture with `node_space` state after 2 iterations (see attachment).

In case where the new reservation is fully inside another reservation,
then everything is OK.
But if the new reservation spans multiple existing reservations then the `end entry record` is not created.
This is because only the newly created `start entry record` is checked.

Easy fix would be to change the if into a loop, for example:

if (placed == true) {
    while((j = node_space[j].next) > 0) {
        if (end_reserve < node_space[j].end_time) {
           //same as above
           break;
        }
    }
    break;
}

2)
You could also change line 612:
        node_space = xmalloc(sizeof(node_space_map_t) *
                             (max_backfill_job_cnt + 3));
To `(max_backfill_job_cnt * 2 + 1)` , since each reservation can add at most two entries (check at line 982 should never execute). At the moment, in a worst case scenario this only checks half of the max_backfill_job_cnt.

NOTE: However this is all based on the assumption, that it is not done on purpose to speed up the calculations and trading some of the accuracy (especially point 2).

Best regards,
Filip Skalski

for better debug