d31965f3 triggered it

When running sacct from a federated client, the db returns jobs for each
cluster with duplicate jobs removed on each cluster. A federated job
could have ran on a different cluster when the before the jobid's
rolled. This patch filters out past old federated jobs and leaves the
newest ones.

With sacct, if multiple clusters were selected then the jobs would be
grouped together by cluster with each cluster's job sorted by submit
time. This patch sorts all of the job by submit time if multiple
clusters are requested.

KNL features: Always keep active and available features in the same order:
    first site-specific features, next MCDRAM modes, last NUMA modes.
bugs 3614, 3679

This change will avoid possible problems in the event that "capmc get_mcdram_cfg"
  fails to return a mcdram_pct value. Never observed, but this will harden the
  code just in case...
bug 3679

Get the federation information from the dbd instead of the controllers.

energy values to uint64_t and storing them in TRES.  What was happening before
was the NO_VAL64 was being added to other NO_VAL64's and creating huge numbers.

bug 3680

bug 3689

bug 3680

Fix to backfill scheduling with respect to QOS and association limits. Jobs
    submitted to multiple partitions are most likley to be effected.
bugs 3680 and 3689

Fix to backfill scheduling with respect to QOS and association limits. Jobs
    submitted to multiple partitions are most likley to be effected.
bugs 3680 and 3689

Bug 3646

are free.

options are mutually exclusive.

Report the job ID using job array formats to better see what is
  happening.

No changes to logic

Coverity CID 45229

Coverity CID 45252

The string "buf" might not be terminated.
Coverity CID 167130, high impact.
Also fixed formatting to match Linux coding standard.

This will be especially valuable when generating reports for a
  federation of clusters that no longer reflects the current
  federation.

Earlier tests would generate errors in a federation as the reports
  would include information from non-local clusters

Bug 3566.

In 6eec8022, the cluster's recv connection is now being destroyed when
the cluster is being destroyed. The problem that showed itself was that
when a remote cluster is removed from the federation, the controller
calls slurmdb_destroy_federation_rec() which destroys the cluster's in
the list. Both the persistent recv thread and the cluster's recv are
pointing to the same thing so when the controller removed the recv
persistent connection the recv thread was pointing to bad memory.