Running jobs can happen out of order.

Original list was being free'd before the copy was.

sbatch|srun --test-only
Bug 3740

Bug 3802

Bug 3641

and steps

Bug 3700

Bug 3699

Bug 3698

Bug 3697

Bug 3667

to validate scontrol --local and --sibling options

Bug 3662

Only used in fed_mgr.c

This will make it easier to add new proto types without having to
modifying protocol_defs.[ch].

Leaving job_lock and job_unlock to be handled by slurmctld_req since
they aren't a "queued" type.

prevent possible memory leak.

All handled in _proc_multi_msg except for sib_job_[un]lock.

This prevents deadlocks when having the fed_job_list_mutex locked higher
up and calling job_completion_logger inside of the locked mutex.

Don't need to have the fed_write_lock when destroying the
persist_conn_server.

Since federated submissions are now asynchronous and because the
working_cluster_rec can be multithreaded, it's better to have the
federadated will_runs in the client. This prevents the deadlocks and
holding up the persistent connections as could happen in the previous
model.

With the change to the asynchronous model, it's better to have the
cluster always get the lock from the origin cluster. Previously, the
origin cluster would try to pick one cluster that could start the job
the soonest and the scenario where there would be only one sibling was
more common. Now that sibling jobs are sent to all clusters this is less
common.

Queue up the fed job completions.

Federated submissions now happen ansynchronously. Sibling jobs are
submitted to the sibling cluster. The sibling cluster queue's up the
request to be handled later when it can get the job write lock. The
sibling cluster submits the job and sends a message back to the origin
cluster which is queued up as well. If the submission failed then the
sibling cluster is removed from the job's active siblings.

The problem was that the origin cluster had to get the internal job
write lock to test and set the fed cluster lock. This would hold up the
persistent connection and get into a dead lock. The solution is create a
separate table for tracking the federated job and the cluster lock which
is controlled by seperate lock.

Plus all communication on the persist connection must be quick. Thus all
communications that need to be modify the actual job need to be put onto
a queue for the scheduler to handle later so that the persistent
connection isn't being held up. The response will be sent back when the
request is processed. This moves to an asynchronous model for
communications between clusters in a federation.