- 16 May, 2012 2 commits
-
-
Danny Auble authored
-
Danny Auble authored
-
- 09 May, 2012 2 commits
-
-
Morris Jette authored
-
Don Lipari authored
The symptom is that SLURM schedules lower priority jobs to run when higher priority, dependent jobs have their dependencies satisfied. This happens because dependent jobs still have a priority of 1 when the job queue is sorted in the schedule() function. The proposed fix forces jobs to have their priority updated when their dependencies are satisfied.
-
- 07 May, 2012 1 commit
-
-
Don Lipari authored
The commit 8b14f388 on Jan 19, 2011 is causing problems with Moab cluster-scheduled machines. Under this case, Moab hands off every job submitted immediately to SLURM which gets a zero priority. Once Moab schedules the job, Moab raises the job's priority to 10,000,000 and the job runs. When you happen to restart the slurmctld under such conditions, the sync_job_priorities() function runs which attempts to raise job priorities into a higher range if they are getting too close to zero. The problem as I see it is that you include the "boost" for zero priority jobs. Hence the problem we are seeing is that once the slurmctld is restarted, a bunch of zero priority jobs are suddenly eligible. So there becomes a disconnect between the top priority job Moab is trying to start and the top priority job SLURM sees. I believe the fix is simple: diff job_mgr.c~ job_mgr.c 6328,6329c6328,6331 < while ((job_ptr = (struct job_record *) list_next(job_iterator))) < job_ptr->priority += prio_boost; --- while ((job_ptr = (struct job_record *) list_next(job_iterator))) { if (job_ptr->priority) job_ptr->priority += prio_boost; } Do you agree? Don
-
- 03 May, 2012 2 commits
-
-
Matthieu Hautreux authored
-
Matthieu Hautreux authored
Here is the way to reproduce it : [root@cuzco27 georgioy]# salloc -n64 -N4 --exclusive salloc: Granted job allocation 8 [root@cuzco27 georgioy]#srun -r 0 -n 30 -N 2 sleep 300& [root@cuzco27 georgioy]#srun -r 1 -n 40 -N 3 sleep 300& [root@cuzco27 georgioy]# srun: error: slurm_receive_msg: Zero Bytes were transmitted or received srun: error: Unable to create job step: Zero Bytes were transmitted or received
-
- 27 Apr, 2012 1 commit
-
-
Danny Auble authored
batch jobs.
-
- 26 Apr, 2012 1 commit
-
-
Morris Jette authored
-
- 25 Apr, 2012 1 commit
-
-
Don Albert authored
Show this HTML in a new window? There is a minor problem with the display of partition names in "sinfo". Without options, the partition name field displays a asterisk "*" at the end of the name of the Default partition. If you specify a formatting option which contains the %P field specifier with a width option (e.g., sinfo -o %8P) the asterisk also is appended to the default partition name. With no width option, the "%P" displays the name based on the full length of the name string, however, no "*" is appended on the default partition name. The attached patch for version 2.4.0-pre4 corrects the problem so that the "*" is correctly appended when %P with no width specifier is used. The patch will also apply to version 2.3.4. -Don Albert-
-
- 24 Apr, 2012 1 commit
-
-
Morris Jette authored
-
- 23 Apr, 2012 2 commits
-
-
Morris Jette authored
-
Par Andersson authored
-
- 20 Apr, 2012 1 commit
-
-
Danny Auble authored
Previously the code would come up with how much memory a PE should have instead of the memory a node should have.
-
- 17 Apr, 2012 1 commit
-
-
Morris Jette authored
Fix sched/wiki2 to support job account name, gres, partition name, wckey, or working directory that contains "#" (a job record separator). Without this patch, the parsing will probably stop once reaching the "#".
-
- 12 Apr, 2012 3 commits
-
-
Danny Auble authored
-
Danny Auble authored
and cons_res or gang scheduling. The slurm.conf is always used for a node configuration in this case, and it ignores what the actual hardware is because in this situation the slurmctld needs to make a bunch of bitmaps before the slurmd's register.
-
Danny Auble authored
-
- 10 Apr, 2012 6 commits
-
-
Morris Jette authored
-
Danny Auble authored
and time limit where it was previously set by an admin.
-
Danny Auble authored
-
Danny Auble authored
slurmdbd accounting and running large amounts of jobs (>50 sec). Job information could be corrupted before it had a chance to reach the DBD.
-
Danny Auble authored
qos.
-
Morris Jette authored
-
- 05 Apr, 2012 1 commit
-
-
Don Lipari authored
While safeguards are in place to prevent unauthorized users from extending the TimeLimit of their running jobs, there were no such restrictions for extending the EndTime. This patch adds the same constraints to modifying EndTime that currently exists for modifying TimeLimit.
-
- 03 Apr, 2012 1 commit
-
-
Morris Jette authored
Add support for new SchedulerParameters of max_depend_depth defining the maximum number of jobs to test for circular dependencies (i.e. job A waits for job B to start and job B waits for job A to start). Default value is 10 jobs.
-
- 02 Apr, 2012 1 commit
-
-
Morris Jette authored
-
- 30 Mar, 2012 1 commit
-
-
Danny Auble authored
-
- 29 Mar, 2012 2 commits
-
-
Morris Jette authored
The problem was conflicting logic in the select/cons_res plugin. Some of the code was trying to get the job the maximum node count in the range while other logic was trying to minimize spreading out of the job across multiple switches. As you note, this problem only happens when a range of node counts is specified and the select/cons_res plugin and the topology/tree plugin and even then it is not easy to reproduce (you included all of the details below). Quoting Martin.Perry@Bull.com: > Certain combinations of topology configuration and srun -N option produce > spurious job rejection with "Requested node configuration is not > available" with select/cons_res. The following example illustrates the > problem. > > [sulu] (slurm) etc> cat slurm.conf > ... > TopologyPlugin=topology/tree > SelectType=select/cons_res > SelectTypeParameters=CR_Core > ... > > [sulu] (slurm) etc> cat topology.conf > SwitchName=s1 Nodes=xna[13-26] > SwitchName=s2 Nodes=xna[41-45] > SwitchName=s3 Switches=s[1-2] > > [sulu] (slurm) etc> sinfo > PARTITION AVAIL TIMELIMIT NODES STATE NODELIST > ... > jkob up infinite 4 idle xna[14,19-20,41] > ... > > [sulu] (slurm) etc> srun -N 2-4 -n 4 -p jkob hostname > srun: Force Terminated job 79 > srun: error: Unable to allocate resources: Requested node configuration is > not available > > The problem does not occur with select/linear, or topology/none, or if -N > is omitted, or for certain other values for -N (for example, -N 4-4 and -N > 2-3 work ok). The problem seems to be in function _eval_nodes_topo in > src/plugins/select/cons_res/job_test.c. The srun man page states that when > -N is used, "the job will be allocated as many nodes as possible within > the range specified and without delaying the initiation of the job." > Consistent with this description, the requested number of nodes in the > above example is 4 (req_nodes=4). However, the code that selects the > best-fit topology switches appears to make the selection based on the > minimum required number of nodes (min_nodes=2). It therefore selects > switch s1. s1 has only 3 nodes from partition jkob. Since this is fewer > than req_nodes the job is rejected with the "node configuration" error. > > I'm not sure where the code is going wrong. It could be in the > calculation of the number of needed nodes in function _enough_nodes. Or > it could be in the code that initializes/updates req_nodes or rem_nodes. I > don't feel confident that I understand the logic well enough to propose a > fix without introducing a regression. > > Regards, > Martin
-
Morris Jette authored
-
- 27 Mar, 2012 2 commits
-
-
Morris Jette authored
When the optional max_time is not specified for --switches=count, the site max (SchedulerParameters=max_switch_wait=seconds) is used for the job. Based on patch from Rod Schultz.
-
Morris Jette authored
Patch by Bill Brophy, Bull.
-
- 26 Mar, 2012 1 commit
-
-
Morris Jette authored
Patch by Don Lipari, LLNL. https://github.com/chaos/slurm/commit/4de11bf0a8cd18207a60e7d3e1fa7a6fde0da431
-
- 21 Mar, 2012 4 commits
-
-
Morris Jette authored
CRAY: Fix support for configuration with SlurmdTimeout=0 (never mark node that is DOWN in ALPS as DOWN in SLURM).
-
Morris Jette authored
-
Morris Jette authored
-
Morris Jette authored
-
- 20 Mar, 2012 2 commits
-
-
Morris Jette authored
Improve support for overlapping advanced reservations. Patch from Bill Brophy, Bull.
-
Morris Jette authored
task/cgroup: minor job step memcg fixes
-
- 18 Mar, 2012 1 commit
-
-
Mark A. Grondona authored
The current task/cgroup memory code writes to force_empty at job step completion and then waits for the release agent to be triggered to remove the memcg. However, force_empty only causes clean cache pages to be dropped from the memcg and does not actually move charges to the parent [1]. This has two unfortunate side-effects. First, pages that can't be dropped by force_empty are in-use and could stay that way indefinitely (e.g. system library that is in-use until just after force_empty completes). Thus, the step memcg never becomes 'empty' and the release agent is not activated. Second, cached pages that can be freed are likely associated with the job itself, and those files and libraries will have to be paged in again for subsequent job steps. In contrast, calling rmdir(2) on a memcg with no active tasks causes *all* current charges to move to parent, which is really what we want in this case. This allows cached libraries and binaries to stay resident and be associated with the job, and also ensures that the step memcg is removed immediately as the job step ends. Thus, this patch replaces the write to force_empty with a call to xcgroup_delete() on the step memcg, which in turn removes the memcg with rmdir(2). The functionality of this patch depends on the previous fix that uses xcgroup_move_process() to move slurmstepd to the root memcg. Otherwise, there will be leftover slurmstepd threads in the job step memcg, and the rmdir will fail with EBUSY. [1] Sec 4.3: http://www.kernel.org/doc/Documentation/cgroups/memory.txt
-