- 03 Jun, 2013 2 commits
-
-
Morris Jette authored
-
Hongjia Cao authored
We're having some trouble getting our slurm jobs to successfully restart after a checkpoint. For this test, I'm using sbatch and a simple, single-threaded executable. Slurm is 2.5.4, blcr is 0.8.5. I'm submitting the job using sbatch: $ sbatch -n 1 -t 12:00:00 bin/bowtie-ex.sh I am able to create the checkpoint and vacate the node: $ scontrol checkpoint create 137 .... time passes .... $ scontrol vacate 137 At that point, I see the checkpoint file from blcr in the current directory and the checkpoint file from Slurm in /var/spool/slurm-llnl/checkpoint. However, when I attempt to restart the job: $ scontrol checkpoint restart 137 scontrol_checkpoint error: Node count specification invalid In slurmctld's log (at level 7) I see: [2013-05-29T12:41:08-07:00] debug2: Processing RPC: REQUEST_CHECKPOINT(restart) from uid=***** [2013-05-29T12:41:08-07:00] debug3: Version string in job_ckpt header is JOB_CKPT_002 [2013-05-29T12:41:08-07:00] _job_create: max_nodes == 0 [2013-05-29T12:41:08-07:00] _slurm_rpc_checkpoint restart 137: Node count specification invalid
-
- 31 May, 2013 1 commit
-
-
Morris Jette authored
Rename slurm_step_ctx_params_t field from "mem_per_cpu" to "pn_min_memory". Job step now accepts memory specification in either per-cpu or per-node basis.
-
- 30 May, 2013 1 commit
-
-
Morris Jette authored
Uninitialized variables resulted in error of "cons_res: sync loop not progressing, holding job #"
-
- 29 May, 2013 1 commit
-
-
jette authored
The most notable problem case is on a cray where a job step specifically requests one or more node that are not the first nodes in the job allocation
-
- 24 May, 2013 2 commits
-
-
Morris Jette authored
-
Morris Jette authored
If set, the job's size component will be based upon not the job size alone, but the job's size divided by it's time limit.
-
- 23 May, 2013 6 commits
-
-
Morris Jette authored
The problem we have observed is the backfill scheduler temporarily gives up its locks (one second), but then reclaims them before the backlog of work completes, basically keeping the backfill scheduler running for a really long time when under a heavy load. bug 297
-
Morris Jette authored
-
Morris Jette authored
Defers (rather than forgets) reboot request with job running on the node within a reservation.
-
Morris Jette authored
-
Morris Jette authored
Defers (rather than forgets) reboot request with job running on the node within a reservation.
-
Danny Auble authored
-
- 22 May, 2013 2 commits
-
-
Danny Auble authored
-
jette authored
-
- 21 May, 2013 1 commit
-
-
Morris Jette authored
Previously nodes in a MAINT reservation would NOT reboot on demand.
-
- 18 May, 2013 1 commit
-
-
Danny Auble authored
all preemptable jobs on the midplane instead of just the ones it needed to.
-
- 16 May, 2013 2 commits
-
-
Morris Jette authored
This bug was introduced in commit f1cf6d2d fix for bug 290
-
Danny Auble authored
-
- 14 May, 2013 1 commit
-
-
Morris Jette authored
-
- 13 May, 2013 1 commit
-
-
Morris Jette authored
Downing the node will kill all jobs allocated to the node, very bad on something like a BlueGene system
-
- 11 May, 2013 1 commit
-
-
Morris Jette authored
This can be especially useful to schedule GPUs. For example a node can be associated with two Slurm partitions (e.g. "cpu" and "gpu") and the partition/queue "cpu" could be limited to only a subset of the node's CPUs, insuring that one or more CPUs would be available to jobs in the "gpu" partition/queue.
-
- 10 May, 2013 1 commit
-
-
Danny Auble authored
-
- 08 May, 2013 2 commits
-
-
David Bigagli authored
-
Danny Auble authored
the node tab and we didn't notice.
-
- 02 May, 2013 3 commits
-
-
Morris Jette authored
The option "interval" was the original parameter, but has not been documented for a year or more.
-
jette authored
Without this change pmdv12 was bound to one CPU and could not use all of the resources allocated to the job step for the tasks that it launches
-
jette authored
This only changes behaviour when the --ntasks option is not used, but the --cpus-per-task option is use
-
- 01 May, 2013 6 commits
-
-
Morris Jette authored
Also add size specification of "%0" to not limit a field size. For example "sacct --format=%0ALL" to print everything.
-
Morris Jette authored
-
Danny Auble authored
-
Morris Jette authored
also "-euidevice sn_single".
-
Morris Jette authored
-
Morris Jette authored
Modify slurmctld data structure locking to interleave read and write locks rather than always favor write locks over read locks.
-
- 30 Apr, 2013 3 commits
-
-
Morris Jette authored
Make timeout configurable at build time by defining SAVE_MAX_WAIT.
-
Olli-Pekka Lehto authored
Dear all, As quick fix, I have put together this script to help manage native and symmetric MPI runs within SLURM. It's a bit bare-bones currently but I needed to get it working quickly :) It does not provide tight integration between the scheduler and MPI daemons and requires a slot on the host, even when running fully on the MIC, so it's really far from an optimal solution but could be a stopgap. It's inspired by the TACC Stampede documentation. They seem to have a similar script in place. It's fairly simple, you provide the names of the MIC binary (with -m) and host binary (with -c). The host MPI/OpenMP parameters are given as usual and the Xeon Phi side parameters as environment variables (MIC_PPN, MIC_OMP_NUM_THREADS). Currently it supports only 1 card per host but extending it should be simple enough. Here are a couple of links to documentation: Our prototype cluster documentation: https://confluence.csc.fi/display/HPCproto/HPC+Prototypes#HPCPrototypes-XeonPhiDevelopment Presentation at the PRACE Spring School in Umeå earlier this week: https://www.hpc2n.umu.se/sites/default/files/1.03%20CSC%20Cluster%20Introduction.pdf Feel free to include this in the contribs -directory. It might need a bit of cleanup though and I don't know when I have the time to do this. I have also added support for TotalView debugger (provided it's installed and configured properly for Xeon Phi usage). Future ideas: For the native MIC client, I've been testing it out a bit and looking at ways to minimize the changes needed for support. The two major challenges seem to be in scheduling and affinity: I think it might be necessary to put it into a specific topology plugin, like the one for BG/Q, but it looks like a lot of work to do that. Best regards, Olli-Pekka
-
Danny Auble authored
-
- 29 Apr, 2013 3 commits
-
-
Morris Jette authored
Avoid placing pending jobs in AdminHold state due to backfill scheduler interactions with advanced reservation. Specifically, if the backfill scheduler tests a pending job can be scheduled after it's advanced reservation ends then the job was assigned a priority of zero (AdminHold).
-
Danny Auble authored
-
Danny Auble authored
undefined variable.
-