Skip to content
GitLab
  1. 10 Apr, 2011 3 commits
    • Moe Jette's avatar
      select/cray: zero reservation ID is not an error · 03f984aa
      Moe Jette authored 
      This avoids meaningless error messages that warn about a zero reservation ID:

 [2011-04-07T15:31:26] _slurm_rpc_submit_batch_job JobId=2870 usec=33390
                       ... a minute later the user decides to scancel the queued job:
 [2011-04-07T15:32:34] error: JobId=2870 has invalid (ZERO) resId
 [2011-04-07T15:32:34] sched: Cancel of JobId=2870 by UID=21770, usec=230

To keep things simple, that test has been removed.

(The patch is in particular also necessary since now job_signal() may trigger
 a basil_release() of a pending job which has no ALPS reservation yet.)
      03f984aa
    • Moe Jette's avatar
      select/cray: release ALPS reservation on termination signals · 12772a3a
      Moe Jette authored 
      On rosa we experienced severe problems when jobs got killed via scancel or
as a result of job timeout. Job cleanup took several minutes, created stray
processes that consumed resources on the slurmd node, keeping the system 
for long spans unable from scheduling.

This problem did not show up on the smaller 2-cabinet XE system (which also
runs a more recent ALPS version). The fix for the problem is to keep new
script lines from starting by sending apkill only after formally releasing
the reservation.

For all signals whose default disposition is to terminate or to dump core,
the reservation is released before signalling the aprun job steps. This
prevents a race condition where further aprun lines get executed while the
apkill of the current aprun line in the job script is in progress.

We did a before/after test on rosa under full load and the problem disappeared.
      12772a3a
    • Moe Jette's avatar
      add testimonial from CSCS · 44bec602
      Moe Jette authored
      44bec602
  3. 09 Apr, 2011 4 commits
  5. 08 Apr, 2011 5 commits
  7. 07 Apr, 2011 12 commits
  9. 06 Apr, 2011 6 commits
  11. 05 Apr, 2011 10 commits