1. 26 Apr, 2012 1 commit
  2. 24 Apr, 2012 1 commit
  3. 23 Apr, 2012 2 commits
  4. 20 Apr, 2012 1 commit
  5. 18 Apr, 2012 1 commit
  6. 17 Apr, 2012 3 commits
  7. 12 Apr, 2012 1 commit
  8. 10 Apr, 2012 4 commits
  9. 09 Apr, 2012 1 commit
  10. 03 Apr, 2012 2 commits
  11. 02 Apr, 2012 2 commits
    • Morris Jette's avatar
      Fix in select/cons_res+topology+job with node range count · cd84134c
      Morris Jette authored
      The problem was conflicting logic in the select/cons_res plugin. Some of the code was trying to get the job the maximum node count in the range while other logic was trying to minimize spreading out of the job across multiple switches. As you note, this problem only happens when a range of node counts is specified and the select/cons_res plugin and the topology/tree plugin and even then it is not easy to reproduce (you included all of the details below).
      
      Quoting Martin.Perry@Bull.com:
      
      > Certain combinations of topology configuration and srun -N option produce
      > spurious job rejection with "Requested node configuration is not
      > available" with select/cons_res. The following example illustrates the
      > problem.
      >
      > [sulu] (slurm) etc> cat slurm.conf
      > ...
      > TopologyPlugin=topology/tree
      > SelectType=select/cons_res
      > SelectTypeParameters=CR_Core
      > ...
      >
      > [sulu] (slurm) etc> cat topology.conf
      > SwitchName=s1 Nodes=xna[13-26]
      > SwitchName=s2 Nodes=xna[41-45]
      > SwitchName=s3 Switches=s[1-2]
      >
      > [sulu] (slurm) etc> sinfo
      > PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
      > ...
      > jkob         up   infinite      4   idle xna[14,19-20,41]
      > ...
      >
      > [sulu] (slurm) etc> srun -N 2-4 -n 4 -p jkob hostname
      > srun: Force Terminated job 79
      > srun: error: Unable to allocate resources: Requested node configuration is
      > not available
      >
      > The problem does not occur with select/linear, or topology/none, or if -N
      > is omitted, or for certain other values for -N (for example, -N 4-4 and -N
      > 2-3 work ok). The problem seems to be in function _eval_nodes_topo in
      > src/plugins/select/cons_res/job_test.c. The srun man page states that when
      > -N is used, "the job will be allocated as many nodes as possible within
      > the range specified and without delaying the initiation of the job."
      > Consistent with this description, the requested number of nodes in the
      > above example is 4 (req_nodes=4).  However, the code that selects the
      > best-fit topology switches appears to make the selection based on the
      > minimum required number of nodes (min_nodes=2). It therefore selects
      > switch s1.  s1 has only 3 nodes from partition jkob. Since this is fewer
      > than req_nodes the job is rejected with the "node configuration" error.
      >
      > I'm not sure where the code is going wrong.  It could be in the
      > calculation of the number of needed nodes in function _enough_nodes.  Or
      > it could be in the code that initializes/updates req_nodes or rem_nodes. I
      > don't feel confident that I understand the logic well enough to propose a
      > fix without introducing a regression.
      >
      > Regards,
      > Martin
      cd84134c
    • Morris Jette's avatar
      Use site maximum for option switch wait time. · 2581fe62
      Morris Jette authored
      When the optional max_time is not specified for --switches=count, the site
      max (SchedulerParameters=max_switch_wait=seconds) is used for the job.
      Based on patch from Rod Schultz.
      2581fe62
  12. 30 Mar, 2012 1 commit
  13. 29 Mar, 2012 2 commits
    • Mark Nelson's avatar
      Added CrpCPUMins to the output of sshare -l for those using hard limit · d1ae3d81
      Mark Nelson authored
      accounting.  Work contributed by Mark Nelson.
      d1ae3d81
    • Morris Jette's avatar
      Fix in select/cons_res+topology+job with node range count · f64b29a2
      Morris Jette authored
      The problem was conflicting logic in the select/cons_res plugin. Some of the code was trying to get the job the maximum node count in the range while other logic was trying to minimize spreading out of the job across multiple switches. As you note, this problem only happens when a range of node counts is specified and the select/cons_res plugin and the topology/tree plugin and even then it is not easy to reproduce (you included all of the details below).
      
      Quoting Martin.Perry@Bull.com:
      
      > Certain combinations of topology configuration and srun -N option produce
      > spurious job rejection with "Requested node configuration is not
      > available" with select/cons_res. The following example illustrates the
      > problem.
      >
      > [sulu] (slurm) etc> cat slurm.conf
      > ...
      > TopologyPlugin=topology/tree
      > SelectType=select/cons_res
      > SelectTypeParameters=CR_Core
      > ...
      >
      > [sulu] (slurm) etc> cat topology.conf
      > SwitchName=s1 Nodes=xna[13-26]
      > SwitchName=s2 Nodes=xna[41-45]
      > SwitchName=s3 Switches=s[1-2]
      >
      > [sulu] (slurm) etc> sinfo
      > PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
      > ...
      > jkob         up   infinite      4   idle xna[14,19-20,41]
      > ...
      >
      > [sulu] (slurm) etc> srun -N 2-4 -n 4 -p jkob hostname
      > srun: Force Terminated job 79
      > srun: error: Unable to allocate resources: Requested node configuration is
      > not available
      >
      > The problem does not occur with select/linear, or topology/none, or if -N
      > is omitted, or for certain other values for -N (for example, -N 4-4 and -N
      > 2-3 work ok). The problem seems to be in function _eval_nodes_topo in
      > src/plugins/select/cons_res/job_test.c. The srun man page states that when
      > -N is used, "the job will be allocated as many nodes as possible within
      > the range specified and without delaying the initiation of the job."
      > Consistent with this description, the requested number of nodes in the
      > above example is 4 (req_nodes=4).  However, the code that selects the
      > best-fit topology switches appears to make the selection based on the
      > minimum required number of nodes (min_nodes=2). It therefore selects
      > switch s1.  s1 has only 3 nodes from partition jkob. Since this is fewer
      > than req_nodes the job is rejected with the "node configuration" error.
      >
      > I'm not sure where the code is going wrong.  It could be in the
      > calculation of the number of needed nodes in function _enough_nodes.  Or
      > it could be in the code that initializes/updates req_nodes or rem_nodes. I
      > don't feel confident that I understand the logic well enough to propose a
      > fix without introducing a regression.
      >
      > Regards,
      > Martin
      f64b29a2
  14. 28 Mar, 2012 1 commit
  15. 27 Mar, 2012 3 commits
  16. 26 Mar, 2012 1 commit
  17. 23 Mar, 2012 1 commit
    • Morris Jette's avatar
      Fix bug in GRES with CPU binding · 4f875d9f
      Morris Jette authored
      Fix bug in allocating GRES that are associated with specific CPUs. In some
      cases the code allocated first available GRES to job instead of allocating
      GRES accessible to the specific CPUs allocated to the job.
      4f875d9f
  18. 22 Mar, 2012 1 commit
  19. 21 Mar, 2012 5 commits
    • Mark A. Grondona's avatar
      Add NEWS items for spank enhancements · 7662d736
      Mark A. Grondona authored
      7662d736
    • Morris Jette's avatar
      change owner of slurmctld and slurmdbd log files · 3470c651
      Morris Jette authored
      Change the owner of slurmctld and slurmdbd log files to the appropriate
      user. Without this change the files will be created by and owned by the
      user starting the daemons (likely user root).
      3470c651
    • Morris Jette's avatar
      CRAY: Fix support for SlurmdTimeout=0 · 4dd9e697
      Morris Jette authored
      CRAY: Fix support for configuration with SlurmdTimeout=0 (never mark
      node that is DOWN in ALPS as DOWN in SLURM).
      4dd9e697
    • Morris Jette's avatar
      Modify the step completion RPC between slurmd and slurmstepd · ed31e6c7
      Morris Jette authored
      in the tightly coupled functions slurmd:stepd_completion and
      slurmstepd:_handle_completion, a jobacct structure is
      send from the main daemon to the step daemon to provide
      the statistics of the children slurmstepd and do the aggregation.
      
      The methodology used to send the structure is the use of
      jobacct_gather_g_{setinfo,getinfo} over a pipe (JOBACCT_DATA_PIPE).
      As {setinfo,getinfo} use a common internal lock and reading
      or writing to a pipe is equivalent to holding a lock, slurmd and
      slurmstepd have to avoid using both setinfo and getinfo over a
      pipe or deadlock situations can occured. For example :
      slurmd(lockforread,write)/slurmstepd(write,lockforread).
      
      This patch remove the call to jobacct_gather_g_setinfo in slurmd
      and the call to jobacct_gather_g_getinfo in slurmstepd ensuring
      that slurmd only do getinfo operations over a pipe and slurmstepd
      only do setinfo over a pipe. Instead jobacct_gather_g_{pack,unpack}
      are used to marshall/unmarshall the data for transmission over the
      pipe.
      Patch by Matthieu Hautreux, CEA.
      
      The patch committed here is a variation on the work by Matthieu.
      Specifically, the logic is added to slurmstepd to read a new format
      of RPC including an RPC version number and buffer with the data
      structure. The slurmd however will not send the RPC in the new format
      until SLURM version 2.5.
      ed31e6c7
    • Morris Jette's avatar
      Modify Makefiles to support Hardening flags · a7e89e72
      Morris Jette authored
      a7e89e72
  20. 20 Mar, 2012 3 commits
  21. 16 Mar, 2012 3 commits