Added "KeepAliveTime" configuration parameter

From Matthieu Hautreux:
TCP_KEEPALIVE addition. No TCP_KEEPALIVE seems to be configured in SLURM TCP
exchanges, thus letting the system potentially deadlocked if a remote host
dissapear and the local host is waiting on a read (the write would result in a
EPIPE or SIGPIPE depending on the masked signals). Adding keepalive with a
relatively large timeout value (5 minutes), could enhance the resilience of
SLURM for unexpected packet/connection loss without too much implication on the
scalability of the solution. The timeout could be configurable in case it is
find too aggresive for particular configurations.

Without this patch, if the first listed partition lacks nodes
with required features the job would be rejected.

While this will validate job at submit time, it results in redundant
looping when scheduling jobs. Working on alternate patch now.

submission.

QoS limits enforcement on the controller side is based on a list of used_limits
per user. When a user is not yet added to the list, which is common when the
controller is restarted and the user has no running jobs, the current logic is
to not check some of the "per user limits" and let the submission succeed.
However, if one of these limits is a zero-valued limit, the check chould
failed as it means that no job should be submitted at all as it would
necessarily result in a crossing of the limit.

This patch ensures that even when a user is not yet present in the per user
used_limits list, the 0-valued limits are correctly treated.

Add PriorityFlags value of "TICKET_BASED".

Correction to CPU allocation count logic in for cores without hyperthreading.

Used to specify the communication protocol to be used for ALPS/BASIL.

allow the use of accounting features like associations, qos and limits but
not keep track of jobs or steps in accounting.

Phase 1 of effort. See "man sbatch" option -a/--array option for details.
Creates job records using sbatch. Reports job arrays using scontrol or
squeue. More work coming soon...

stack.

If sched/backfill starts a job with a QOS having NO_RESERVE and not job
time limit, start it with the partition time limit (or one year if the
partition has no time limit) rather than NO_VAL (140 year time limit);

If a standby job, which in this
case has the NO_RESERVE flag set, is submitted
without a time limit, and is backfilled, it
will get an EndTime waaayyyy into the future.

JobId=99 Name=cmdll
   UserId=eckert(1043) GroupId=eckert(1043)
   Priority=12083 Account=sa QOS=standby
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
   RunTime=00:00:14 TimeLimit=12:00:00 TimeMin=N/A
   SubmitTime=2012-12-20T11:49:36 EligibleTime=2012-12-20T11:49:36
   StartTime=2012-12-20T11:49:44 EndTime=2149-01-26T18:16:00

so I looked at the code in /src/plugins/sched/backfill:

                if (job_ptr->start_time <= now) {
                        int rc = _start_job(job_ptr, resv_bitmap);
                        if (qos_ptr && (qos_ptr->flags & QOS_FLAG_NO_RESERVE)){
                                job_ptr->time_limit = orig_time_limit;
                                job_ptr->end_time = job_ptr->start_time +
                                                    (orig_time_limit * 60);

Using the debugger I found that if the job does not have a specified
time limit, the job_ptr->time_limit is equal to NO_VAL when it hits
this code.

2.5.

Identify node states on which HealthCheckProgram should be executed.

slurm.conf with NodeAddr's signals going to a step could be handled
incorrectly.

would of also killed the allocation.

This is a variation of "slurm_load_nodes", but accepts a node name argument, potentially resultingin substantial performance improvement for "sinfo --nodes=NAME".

This is a variation of "slurm_load_jobs", but accepts a user ID argument,
potentially resulting in substantial performance improvement.

to make one job run.

-N1 -n#.

This adds the fields to the data structures and configuration file
to be used later for executing a program at the beginning and end
of each reservation.