The previous commit obviously fixed a problem, but introduced a different
set of problems. This will be pursued later, perhaps in version 16.05.

With some configurations and systems, errors of the following sort were
occuring:
task/cgroup: task[1] infinite loop broken while trying to provision compute elements using block
task/cgroup: task[1] unable to set taskset '0x0'

Bug 1706

bug 2070

burst_buffer/cray - If the pre-run operation fails then don't issue
    duplicate job cancel/requeue unless the job is still in run state. Prevents
    jobs hung in COMPLETING state.
bug 2587

Running "scontrol reconfig" releases resources for jobs waiting for
  the completion of Node Health Check so that other jobs can run.
  Cray says to always wait for NHC to complete, but in extreme
  cases that can be 2 hours, during which the entire resource
  allocation for a job may be unusable. Per advice from NERSC,
  the logic to release resources is unchanged, but logging is
  added here.

isn't kept up to date in the cache.

as will.

Conflicts:
	src/plugins/select/cons_res/job_test.c

Fix gang scheduling resource selection bug which could prevent multiple jobs
    from being allocated the same resources. Bug was introduced in 15.08.6,
    commit 44f491b8

Also ensure empty (0-length) files are handled properly.
Remove a stray exit(1) call from _rpc_file_bcast() to avoid
slurmd exiting on malformed data.

Here's how to reproduce on smd-server with 2 sockets, 6 cores per
socket and 2 threads per core, just run the following command line
3 times in quick succession (all active at the same time):
srun --cpus-per-task=4 -m block sleep 30
What was happening is the first job would be allocated cores 0+1
The second job would be allocated cores 2+3
The thrid job would test use of cores 0-3 then exit because the
 job only needs 4 CPUs. The resulting core binding would include
 NO CPUs. The new logic tests that the core being considered for
 use actually has some resources available to the job before
 updating the counter which is being tested against the needed
 CPU counter.

Specifically add the HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM flag when
loading configuration from HWLOC library. Previous logic in
task/cgroup did not do this, which was different behaviour from
how slurmd gets configuration information. Here's the HWLOC
documentation:
HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM
Detect the whole system, ignore reservations and offline settings.
Gather all resources, even if some were disabled by the administrator.
For instance, ignore Linux Cpusets and gather all processors and memory
nodes, and ignore the fact that some resources may be offline.

Without this flag, I was rarely observing a bad core count, which
resulted in the logic layout out tasks wrong and generating an error:
task/cgroup: task[0] infinite loop broken while trying to provision compute elements using cyclic

bug 2502

With bcast split into its own directory -lz should not be
required throughout.

This reverts commit e7981406.

Remove unused struct and macro from file_bcast.h.

Free file_bcast_info_t to prevent leak.

1) Add a new global file_bcast_list to store info on in-progress file
   transfers, cache FD there rather than reopening the file for every block.
2) Restructure security mechanisms. First block will fork() and open the
   file, and pass the FD back to the thread. Thread then registers this file
   transfer in the file_bcast_list. Split fork() stuff into
   _file_bcast_register_file to keep _rpc_file_bcast readable.
3) Successive blocks are handled within the thread. Security is handled by
   matching uid and file name to existing file transfer.

TODO:

1) Write transfer cleanup function to remove stalled transfers.
2) Use mmap for file output.
3) Allow for parallel block transfer. Current code assumes blocks will always
   arrive in order. Out of order blocks will result in corrupted output.
   (sbcast currently prevents this by requiring each message to be ack'd
   before continuing, but at a likely severe performance penalty.)
4) Add stats on receive side.

This reverts commit 8c8c3407488fe3f0a552d2359ef5b487330ee8ba.

Thread-only isn't portable, need to use fork() on first block to ensure
file security and containers are handled correctly.

Fix bad cast in 3a604563, and update pct to 64-bits to prevent
truncation of intermediate value (pct * 100).

Update srun and sbcast man pages and help messages for compression
  library argument.

Relies on Linux-specific behavior of setfsuid/gid, so disable on other
platforms for now.

Fix linking issues to only have the compression libs link to file_bcast which intern get pulled in my anyone linking to the .la.

Move compression stuff over to the file_bcast lib, so we only have to link directly with that and not pull libs all over the place.

- Fix issue where max_out calculation for zlib was incorrect - use
  deflateBound to properly calculate required buffer size, 1024 is
  not sufficient padding for uncompressable input.

- Work towards sending file offsets across the wire in preparation
  for mmap'd output for lz4.

Rather than compressing at most block_len into a message, compress up to
(10 * block_len) into a single message.

10x arbitrarily chosen to mitigate buffer issues when uncompresssing in
slurmd.

Testing against a 100MB zero file, this reduces the messages required from
13 to 2.

Regression in last commit required zlib to send each chunk as a
separate block, rather than packing multiple chunks per block.