This reording of some code results in cleaner logic

collapse the scheduling table when possible to reduce the
number of time slots to check for pending jobs. This should
improve performance considerably.

Previous logic was sometimes building incomplete map

When a decision is made to start a job, if for some
reason that job's start failed, the backfill scheduler
would previously just exit. With this change, it logs
the event and reserves the resources expected to be
used and continues down the job queue.

This change prevents creation of some back-to-back records with
the same resources, but different times.

No change in logic

Improved logging of backfill scheduling actions
Better handling of backfill_resolution logic to avoid creating
   some records that are not needed
Avoid creating some backfill scheduling maps with zero duration
The net effect should be slightly improved performance with no
   significant difference in action

Update slurm.conf man page for DebugFlag BackfillMap. This should be
considered part of commit 3c2bffb6

Add DebugFlag of BackfillMap. Previously a DebugFlag value of Backfill
logged information about what it was doing plus a map of expected resouce
use in the future. Now that very verbose resource use map is only logged
with a DebugFlag value of BackfillMap

Log not only the count of jobs tested since the last time locks
were released, but also the total job count since the backfill
scheduler started.

Remove duplicate backfill scheduling tests. For example there is
no need to test if a job can be started if the only difference
from the previous test involves nodes in other partitions that
can not be used by the job we are trying to start.

The backfill scheduler was always reporting the time that a job
was being considered as NOW rather than the time that was really
being considered.

decreases and total is less than in use.

This is a correction to commit 308b7432

Improve how failures in slurmd/slurmstepd communications are logged.

mail messages for job array events print now use the job ID using the
format "#_# (#)" rather than just the internal job ID.

This will help limit damage from two active primary slurmctld
(split brain problem).

it is already running.

Duplicate triggers are not not allowed

Job profiling leaves a file open

job is JOB_COMPLETING or already pending.

last epilog completes, either slurmd epilog or slurmctld epilog,
whichever comes last.

don't clear the dependency if the job is completing.

(Also remove extra pending check, no reason to check it twice ;))

If the backup slurmctld assumes primary status, then do NOT purge any
job state files (batch script and environment files) but if any attempt
is made to re-use them consider this a fatal error. It may indicate that
multiple primary slurmctld daemons are active (e.g. both backup and primary
are functioning as primary and there is a split brain problem).

Replace printing of job_id using %d with %u

Test time when job_state file was written to detect multiple primary
slurmctld daemons (e.g. both backup and primary are functioning as
primary and there is a split brain problem).

Signed-off-by: Danny Auble <da@schedmd.com>