1. 09 May, 2016 2 commits
    • Morris Jette's avatar
      update dw_wlm_cli emulator · 6daf89d1
      Morris Jette authored
      6daf89d1
    • Morris Jette's avatar
      bb/cray: add support for "equalize_fragments" · 5d67e075
      Morris Jette authored
      burst_buffer/cray - Add support for rounding up the size of a buffer reqeust
          if the DataWarp configuration "equalize_fragments" is used. That option
          can significantly increase the size of a burst buffer allocation in order
          to create equal sized buffers on all server nodes for performance reasons.
          Support current only provided by Cray for job buffers, not persistent
          burst buffers.
      5d67e075
  2. 06 May, 2016 6 commits
    • Morris Jette's avatar
      Merge branch 'slurm-16.05' · 69b567fc
      Morris Jette authored
      69b567fc
    • Morris Jette's avatar
      Merge branch 'slurm-15.08' into slurm-16.05 · eb888964
      Morris Jette authored
      eb888964
    • John Thiltges's avatar
      Fix for slurmstepd setfault · db0fe22e
      John Thiltges authored
      With slurm-15.08.10, we're seeing occasional segfaults in slurmstepd. The logs point to the following line: slurm-15.08.10/src/slurmd/slurmstepd/mgr.c:2612
      
      On that line, _get_primary_group() is accessing the results of getpwnam_r():
          *gid = pwd0->pw_gid;
      
      If getpwnam_r() cannot find a matching password record, it will set the result (pwd0) to NULL, but still return 0. When the pointer is accessed, it will cause a segfault.
      
      Checking the result variable (pwd0) to determine success should fix the issue.
      db0fe22e
    • Morris Jette's avatar
      Document lack of current support for KNL quad/SNC2 · 7246b955
      Morris Jette authored
      Note that Slurm can not support heterogenous core counts for each
      NUMA nodes.
      bug 2704
      7246b955
    • Marco Ehlert's avatar
      Correct partition MaxCPUsPerNode enforcement · 70aafa68
      Marco Ehlert authored
      I would like to mention a problem which seems to be a calculation bug of
      used_cores in slurm version 15.08.7
      
      If a node is divided into 2 partitions using MaxCPUsPerNode like this
      slurm.conf configuration
      
          NodeName=n1 CPUs=20
          PartitionName=cpu NodeName=n1    MaxCPUsPerNode=16
          PartitionName=gpu NodeName=n1    MaxCPUsPerNode=4
      
      I run into a strange scheduling situation.
      The situation occurs after a fresh restart of the slurmctld daemon.
      
      I start jobs one by one:
      
      case 1
          systemctl restart slurmctld.service
          sbatch -n 16 -p cpu cpu.sh
          sbatch -n 1  -p gpu gpu.sh
          sbatch -n 1  -p gpu gpu.sh
          sbatch -n 1  -p gpu gpu.sh
          sbatch -n 1  -p gpu gpu.sh
      
          => Problem now: The gpu jobs are kept in PENDING state.
      
      This picture changes if I start the jobs this way
      
      case 2
          systemctl restart slurmctld.service
          sbatch -n 1  -p gpu gpu.sh
          scancel <gpu job_id>
          sbatch -n 16 -p cpu cpu.sh
          sbatch -n 1  -p gpu gpu.sh
          sbatch -n 1  -p gpu gpu.sh
          sbatch -n 1  -p gpu gpu.sh
          sbatch -n 1  -p gpu gpu.sh
      
      and all jobs are running fine.
      
      By looking into the code I figured out a wrong calculation of 'used_cores' in
      function _allocate_sc()
      
      plugins/select/cons_res/job_test.c
      
      _allocate_sc(...)
      ...
               for (c = core_begin; c < core_end; c++) {
                       i = (uint16_t) (c - core_begin) / cores_per_socket;
      
                       if (bit_test(core_map, c)) {
                               free_cores[i]++;
                               free_core_count++;
                       } else {
                               used_cores[i]++;
                       }
                       if (part_core_map && bit_test(part_core_map, c))
                               used_cpu_array[i]++;
      
      This part of code seems to work only if the part_core_map exists for a
      partition or on a completly free node. But in case 1 there is no
      part_core_map for gpu created yet. Starting a gpu  the core_map contains
      4 cores left from the cpu job. Now all non free cores of the cpu partion
      are counted as used cores in the gpu partition and this condition will
      match in the next code parts
      
          free_cpu_count + used_cpu_count >  job_ptr->part_ptr->max_cpus_per_node
      
      what is definitely wrong.
      
      As soon as a part_core_map appears, means a gpu job was started on a free
      node (case 2) then there is no problem at all.
      
      To get case 1 work I changed the above code to the following and all works
      fine:
      
               for (c = core_begin; c < core_end; c++) {
                       i = (uint16_t) (c - core_begin) / cores_per_socket;
      
                      if (bit_test(core_map, c)) {
                               free_cores[i]++;
                               free_core_count++;
                       } else {
                           if (part_core_map && bit_test(part_core_map, c)){
                               used_cpu_array[i]++;
                               used_cores[i]++;
                           }
                       }
      
      I am not sure this code change is really good, but it fixes my problem.
      70aafa68
    • Brian Christiansen's avatar
      Update configure script for 17.02 · 34db1cce
      Brian Christiansen authored
      34db1cce
  3. 05 May, 2016 12 commits
  4. 04 May, 2016 12 commits
  5. 03 May, 2016 8 commits