- 17 Oct, 2014 6 commits
-
-
Morris Jette authored
-
Morris Jette authored
-
David Bigagli authored
-
David Bigagli authored
-
Morris Jette authored
Correct tracking of licenses for suspended jobs on slurmctld reconfigure or restart. Previously licenses for suspended jobs were not counted, so the license count could be exceeded with those jobs get resumed.
-
Danny Auble authored
commit a9dc50d4.
-
- 16 Oct, 2014 5 commits
-
-
Brian Christiansen authored
-
Danny Auble authored
to commit a9dc50d4.
-
Morris Jette authored
-
Morris Jette authored
Refine commit 5f89223f based upon feedback from David Gloe: * It's not only MPI jobs, but anything that uses PMI. That includes MPI, shmem, etc, so you may want to reword the error message. * I added the terminated flag because if multiple tasks on a node exit, you would get an error message from each of them. That reduces it to one error message per node. Cray bug 810310 prompted that change. * Since we're now relying on --kill-on-bad-exit, I think we should update the Cray slurm.conf template to default to 1 (set KillOnBadExit=1 in contribs/cray/slurm.conf.template). bug 1171
-
Morris Jette authored
Treat Cray MPI job calling exit() without mpi_fini() as fatal error for that specific task and let srun handle all timeout logic. Previous logic would cancel the entire job step and srun options for wait time and kill on exit were ignored. The new logic provides users with the following type of response: $ srun -n3 -K0 -N3 --wait=60 ./tmp Task:0 Cycle:1 Task:2 Cycle:1 Task:1 Cycle:1 Task:0 Cycle:2 Task:2 Cycle:2 slurmstepd: step 14927.0 task 1 exited without calling mpi_fini() srun: error: tux2: task 1: Killed Task:0 Cycle:3 Task:2 Cycle:3 Task:0 Cycle:4 ... bug 1171
-
- 15 Oct, 2014 9 commits
-
-
Morris Jette authored
This fixes a race condition if the slurmctld needed to power up a node shortly after startup. Previously it would execute the ResumeProgram twice for effected nodes.
-
Morris Jette authored
Without this change, a node in the cloud that failed to power up, would not have its NoResponding flag cleared, which would prevent its later use. The NoResponding flag is now cleared when manuallly when the node is modified to PowerDown.
-
Morris Jette authored
If a batch job launch to the cloud fails, permit an unlimited number of job requeues. Previously the job would abort on the second launch failure.
-
Nicolas Joly authored
This reverts commit 4d03d0b4. Make sure the correct Author is attributed here.
-
Danny Auble authored
This reverts commit 1891936e.
-
Danny Auble authored
This has apparently been broken from the get go. This fixes bug 1172. test21.22 should be updated to test the dump and load of a file that is generated.
-
Danny Auble authored
since this represents the user could be exaggerating their system.
-
Danny Auble authored
-
Danny Auble authored
using --ntasks-per-node. This is related to bug 1145. What was happening is all the cpus were allocated on one socket instead of a cyclic method. While this is allowed it is strange and resulted in this bug. There appears to be a different bug as to why the tasks were laid out in a block fashion in the first place.
-
- 14 Oct, 2014 8 commits
-
-
Danny Auble authored
with no way to get them out. This fixes bug 1134. It is advised the pro/epilog to call xtprocadmin in the script instead of returning a non-zero exit code.
-
Brian Christiansen authored
The job could have been purged from a short MinJobAge and the trigger would then point to an invalid job. Bug #1144
-
Danny Auble authored
-
Morris Jette authored
Note that PlugStackConfig defaults to plugstack.conf in the same directory as slurm.conf. The added logic tests if the file actually exists (using stat) and if not found then do not fork/exec slurmstepd to invoke the spank prolog/epilog. This saves about 14msec on startup and 14msec on shutdown if no spank plugins are configured. It also eliminates some possible failures (e.g. if fork() fails, or the slurmstepd processes can not exec()). This logic also caches the PlugStackConfig value and reads it again on reconfigure, but avoid reading the value for each job. bug 982
-
Morris Jette authored
Add "void" argument to a function and rename a local function to have a prefix of "_"
-
Nicolas Joly authored
-
Danny Auble authored
9b00f12c
-
Nicolas Joly authored
Signed-off-by:
Danny Auble <da@schedmd.com>
-
- 11 Oct, 2014 3 commits
-
-
Morris Jette authored
if a node is down, then permit setting its state to power down, which causes the SuspendProgram to run and set the node state back to cloud.
-
Morris Jette authored
If a node is powered down, then do not power it up on slurmctld restart.
-
Morris Jette authored
The power up/down request only takes effect after the ResumeTimeout or SuspendTimeout is reached in order to avoid a race condition.
-
- 10 Oct, 2014 9 commits
-
-
Danny Auble authored
-
Brian Christiansen authored
Bug #1143
-
Danny Auble authored
-
Dorian Krause authored
This commit fixes a bug we observed when combining select/linear with gres. If an allocation was requested with a --gres argument an srun execution within that allocation would stall indefinitely: -bash-4.1$ salloc -N 1 --gres=gpfs:100 salloc: Granted job allocation 384049 bash-4.1$ srun -w j3c017 -n 1 hostname srun: Job step creation temporarily disabled, retrying The slurmctld log showed: debug3: StepDesc: user_id=10034 job_id=384049 node_count=1-1 cpu_count=1 debug3: cpu_freq=4294967294 num_tasks=1 relative=65534 task_dist=1 node_list=j3c017 debug3: host=j3l02 port=33608 name=hostname network=(null) exclusive=0 debug3: checkpoint-dir=/home/user checkpoint_int=0 debug3: mem_per_node=62720 resv_port_cnt=65534 immediate=0 no_kill=0 debug3: overcommit=0 time_limit=0 gres=(null) constraints=(null) debug: Configuration for job 384049 complete _pick_step_nodes: some requested nodes j3c017 still have memory used by other steps _slurm_rpc_job_step_create for job 384049: Requested nodes are busy If srun --exclusive would have be used instead everything would work fine. The reason is that in exclusive mode the code properly checks whether memory is a reserved resource in the _pick_step_node() function. This commit modifies the alternate code path to do the same.
-
Morris Jette authored
-
Brian Christiansen authored
-
Danny Auble authored
(i.e ArchiveJobs PurgeJobs). This is only a cosmetic change.
-
Nicolas Joly authored
on slurmdbd startup.
-
Danny Auble authored
-