This patch has fixed the problem to me. We are going to do some more
verification later today and update.
But I would appreciate if somebody else can test it as well.

Signed-off-by: Danny Auble <da@schedmd.com>

There was a list of pending pack job records under consideration
  for scheduling by the backfill plugin that was not being cleared
  between interations of the backfill scheduler resulting in various
  scheduling anomalies.
bug 4371, 4400

An abort was triggered here due to a pack job start failure.

code dealing with how we need to keep track of it.

Bug 4405

For heterogeneous job steps, the srun --open-mode option default value will
    be set to "append".

Previous logic would fail if more than 2 pack groups implicitly
  specified in srun command.

Previous logic would delay the initiation of pack jobs until all
  components were submitted. The new logic will defer pack job
  scheduling based upon a new "pack_job_offset" field in the
  job submit request and NOT set a begin_time in the future.
  This will eliminate the pack job scheduling reason value
  of "BEGIN_TIME".
bugs 4369, 4400

Fix for bug introduced in commit 9e0b976a
bug 4400

The regex in x11_set_xauth() did not match FQDNs because it was missing
a dot.

Bug 4398.

If a node's physical Boards, Sockets, Cores, Threads, etc.
  differ from configuration log using error() rather than info().
bug 4394

Add SchedulerParameters=whole_pack configuration parameter. If set, then
    hold, release and cancel operations on any component of a heterogeneous job
    will be applied to all components.
bug 4374

Bug 4393.

Running "scontrol reconfig" re-initialized some arrays/lists without
  clearing the previous values, resulting in a memory leak.

As reported by Valgrind

Previous logic would continuously report command not responding,
  but not exit the wait loop. Only test15.# checked so far

Bug 4366

bug 4366

Coverity CID 179255

Fix logic so that we can find all components of a pack job if the
  submit fails. Populate the "pack_job_list" field on a pack job
  leader if some later component can't be submitted.

Correct printing error type based upon errno rather than returned rc.

If PrologSlurmctld fails for pack job leader then kill all
    components of the job.
bug 4379

Add SLURM_PACK_JOB_NODELIST to PrologSlurmctld and EpilogSlurmctld
    environment.
bug 4379

bug 4370

Prevent scheduling deadlock with multiple components of heterogeneous job
    in different partitions (i.e. one heterogeneous job component is higher
    priority in one partition and another component is lower priority in a
    different partition).
bug 4370

Issue could be reproduced by restarting slurmctld after a heterogeneous job
finished, but before MinJobAge time passed. Since the pack_job_list
job_record memeber wasn't saved/loaded to/from the job_state, the function
_validate_pack_jobs() is responsible for rebuilding the pack_job_list. Issue
was that the function was skiping the rebuild work for finished jobs, thus
other functions like the thread responsible for purging old jobs was failing
to iterate over a NULL pack_job_list which was never rebuilt.

Bug 4383.

If run from srun and lua job submit plugin sets environment, slurmctld
crashes.

Bug#4247

From within slurm_job_submit():
	job_desc.pack_job_offset

From within slurm_job_modify():
	job_rec.pack_job_id
	job_rec.pack_job_id_set
	job_rec.pack_job_offset

Bug 4372.

bug 4339

added some additional checks to prevent segfaults in some basic situations.
Bug 4247

bug 4368

Add SLURM_PACK_JOB_ID and SLURM_PACK_JOB_OFFSET to PrologSlurmctld and
  EpilogSlurmctld environment
bug 4379

Avoid srun abort trying to run on heterogeneous job component that has
    ended.
bug 4366

bug 4374

Do so even if pack-group 0 is completed, so long as not all components are completed
bug 4374