- 26 Aug, 2015 5 commits
-
-
Morris Jette authored
Prevent job array task ID from being reported as NO_VAL if last task in the array gets requeued. The problem is that when that task starts, the task bitmap entry for it stays set, but the task counter gets decremented. If that job then gets requeued, under some conditions a failure to schedule it results in the array_task_id in the job record getting set to NO_VAL. Then when building the job info to report for squeue/scontrol, the string showing the pending task ID's is not rebuilt due to that counter being zero. All indications are that the job runs fine, only the information reported to squeue/scontrol is wrong. bug 1790
-
Danny Auble authored
-
Danny Auble authored
-
Danny Auble authored
-
Danny Auble authored
accounting correctly to avoid meaningless errors about overflow.
-
- 25 Aug, 2015 4 commits
-
-
Brian Christiansen authored
-
Brian Christiansen authored
Bug 1873
-
Danny Auble authored
intended for the head of the tree.
-
Danny Auble authored
or binary.
-
- 24 Aug, 2015 1 commit
-
-
Danny Auble authored
wouldn't get the correct protocol version to launch a step.
-
- 22 Aug, 2015 1 commit
-
-
Morris Jette authored
-
- 21 Aug, 2015 6 commits
-
-
Brian Christiansen authored
Bug 1831
-
Brian Christiansen authored
Bug 1869
-
David Bigagli authored
-
Morris Jette authored
Fix gang scheduling/preemption issue that could cancel job at startup. I have not been able to reproduce the reported problem, but this should prevent the reported problem. bug 1880
-
Morris Jette authored
-
Danny Auble authored
-
- 20 Aug, 2015 2 commits
-
-
David Bigagli authored
This reverts commit 406cb529. Conflicts: NEWS This works ok up until 1048576M then it will make the divisor 1048576 and when it goes to divide it only does it once instead of 2 times returning 1G instead of 1T. We have decided to wait until 15.08 for a no convert option to be added.
-
David Bigagli authored
This reverts commit 406cb529. Conflicts: NEWS This works ok up until 1048576M then it will make the divisor 1048576 and when it goes to divide it only does it once instead of 2 times returning 1G instead of 1T. We have decided to wait until 15.08 for a no convert option to be added.
-
- 19 Aug, 2015 1 commit
-
-
Brian Christiansen authored
-
- 14 Aug, 2015 4 commits
-
-
Brian Christiansen authored
Bug 858
-
Brian Christiansen authored
Fix case where job would get the wrong cpu count when using --ntasks-per-core and --cpus-per-task together. Bug 1370
-
Daniel Ahlin authored
We are using a non-default AuthInfo configuration and based on log-messages we see I believe this is not properly handled in certain parts of the code. Typical log message: Aug 12 17:06:15 t02n20 slurmd[27001]: error: Munge encode failed: Failed to access "/var/run/munge/munge.socket.2": No such file or directory Aug 12 17:06:15 t02n20 slurmd[27001]: error: Creating authentication credential: Socket communication error Aug 12 17:06:15 t02n20 slurmd[27001]: error: stepd_connect to 3165.0 failed: Protocol authentication error Aug 12 17:06:15 t02n20 slurmd[27001]: error: If munged is up, restart with --num-threads=10
-
Morris Jette authored
Unfortunately the reservation's core_bitmap is a global bitmap and the nodes in the system may have changed in terms of their node count or nodes have been added or removed. Make best effort to rebuild the reservation's core_bitmap on the limited information currently available. The specific cores might change, but this logic at least leaves their count constant and uses the same nodes bug 1850
-
- 11 Aug, 2015 2 commits
-
-
David Bigagli authored
-
Brian Christiansen authored
Bug 1770
-
- 06 Aug, 2015 1 commit
-
-
Danny Auble authored
thing (-F exclusive) instead of -F share. bug 1849
-
- 05 Aug, 2015 1 commit
-
-
Morris Jette authored
This reverts part of commit c2fbf88f see bug 1846
-
- 04 Aug, 2015 2 commits
-
-
David Bigagli authored
-
Morris Jette authored
Fix scheduling anomaly with job arrays submitted to multiple partitions, jobs could be started out of priority order. bug 1892
-
- 03 Aug, 2015 3 commits
-
-
Danny Auble authored
portion of a job array is pending after it's initial start in the database it could create another row for the remain array instead of using the already existing row. bug 1840
-
Danny Auble authored
-
Dorian Krause authored
This commit adds a new field free_mem to the node_info struct and augments the 15.08 RPC protocol to exchange this information as part of the ping and registration status messages. The free memory is taken from the sysinfo() information on systems that support this function. Overall, the implementation sticks to the way cpu_load (the 5 min load average) is handled.
-
- 31 Jul, 2015 1 commit
-
-
David Bigagli authored
-
- 28 Jul, 2015 5 commits
-
-
Brian Christiansen authored
Fix segfault in Bluegene setups where RebootQOSList is defined in bluegene.conf and accounting is not setup.
-
Nathan Yee authored
Read in correct number of nodes from SLURM_HOSTFILE when specifying nodes and --distribution=arbitrary. Bug 1808
-
Brian Christiansen authored
From eee63695
-
Brian Christiansen authored
Bug 1810
-
Thomas Cadeau authored
-
- 27 Jul, 2015 1 commit
-
-
Brian Christiansen authored
Bug 1819 Composite indexes search left to right. E.g. an index of (inx1, inx2, inx3) will search from left to right. inx2 can't be used in a where statement by itself, it requires inx1 to be present (inx3 can be optional). For the rollup index having time_end first speeds up the below query. The actual rollup queries still benefit from the original rollup index. sacct -S 07/22-09:41:36 -E 07/22-09:42:37 -i 1-4 -ojobid,start,end,nnodes,nodelist -n -a: mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where ((t1.nodes_alloc between 1 and 4)) && ((t1.time_eligible < 1437550957 && (t1.time_end >= 1437550896 || t1.time_end = 0))) group by id_job, time_submit desc; +----+-------------+-------+--------+---------------+---------+---------+------------------------+--------+----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+---------------+---------+---------+------------------------+--------+----------------------------------------------+ | 1 | SIMPLE | t1 | ALL | id_job,rollup | NULL | NULL | NULL | 120953 | Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+---------------+---------+---------+------------------------+--------+----------------------------------------------+ 3 rows in set (0.00 sec) mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where ((t1.nodes_alloc between 1 and 4)) && ((t1.time_eligible < 1437550957 && (t1.time_end >= 1437550896 || t1.time_end = 0))) group by id_job, time_submit desc; +----+-------------+-------+--------+-----------------------+---------+---------+------------------------+------+---------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+-----------------------+---------+---------+------------------------+------+---------------------------------------------------------------------+ | 1 | SIMPLE | t1 | range | id_job,rollup,rollup2 | rollup2 | 8 | NULL | 6 | Using index condition; Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+-----------------------+---------+---------+------------------------+------+---------------------------------------------------------------------+ 3 rows in set (0.00 sec) rollup: mysql> explain select job.job_db_inx, job.id_job, job.id_assoc, job.id_wckey, job.array_task_pending, job.time_eligible, job.time_start, job.time_end, job.time_suspended, job.cpus_alloc, job.cpus_req, job.id_resv, SUM(step.consumed_energy) from compy_job_table as job left outer join compy_step_table as step on job.job_db_inx=step.job_db_inx and (step.id_step>=0) where (job.time_eligible < 1420102800 && (job.time_end >= 1420099200 || job.time_end = 0)) group by job.job_db_inx order by job.id_assoc, job.time_eligible; +----+-------------+-------+-------+--------------------------------------------------------------------------------+---------+---------+---------------------------+------+--------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+-------+--------------------------------------------------------------------------------+---------+---------+---------------------------+------+--------------------------------------------------------+ | 1 | SIMPLE | job | range | PRIMARY,id_job,rollup,rollup2,wckey,qos,association,array_job,reserv,sacct_def | rollup | 4 | NULL | 1 | Using index condition; Using temporary; Using filesort | | 1 | SIMPLE | step | ref | PRIMARY | PRIMARY | 4 | slurm_1412.job.job_db_inx | 1 | Using where | +----+-------------+-------+-------+--------------------------------------------------------------------------------+---------+---------+---------------------------+------+--------------------------------------------------------+ 2 rows in set (0.01 sec) A plain sacct is sped up by moving time_end into the middle of the index (ex. id_user, time_end, time_eligible). sacct_def is for sacct calls with a state specified, sacct_def2 is for a plain sacct call. plain sacct: mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where (t1.id_user='1003') && ((t1.time_end >= 1437548400 || t1.time_end = 0)) group by id_job, time_submit desc; +----+-------------+-------+--------+------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ | 1 | SIMPLE | t1 | ref | id_job,sacct_def | sacct_def | 4 | const | 60476 | Using index condition; Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ 3 rows in set (0.00 sec)o mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where (t1.id_user='1003') && ((t1.time_end >= 1437548400 || t1.time_end = 0)) group by id_job, time_submit desc; +----+-------------+-------+--------+-------------------------------------+------------+---------+------------------------+------+--------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+-------------------------------------+------------+---------+------------------------+------+--------------------------------------------------------+ | 1 | SIMPLE | t1 | range | id_job,rollup2,sacct_def,sacct_def2 | sacct_def2 | 8 | NULL | 68 | Using index condition; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+-------------------------------------+------------+---------+------------------------+------+--------------------------------------------------------+ 3 rows in set (0.00 sec) Adding the sacct_def2 index order didn't affect other queries: sacct -s CA,CD,F,R: mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where (t1.id_user='1003') && ((t1.state='4' && (t1.time_end && (t1.time_end >= 1438028802))) || (t1.state='3' && (t1.time_end && (t1.time_end >= 1438028802))) || (t1.state='5' && (t1.time_end && (t1.time_end >= 1438028802))) || (t1.time_start && ((!t1.time_end && t1.state=1) || (1438028802 between t1.time_start and t1.time_end)))) group by id_job, time_submit desc; +----+-------------+-------+--------+-------------------------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+-------------------------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ | 1 | SIMPLE | t1 | ref | id_job,rollup2,sacct_def,sacct_def2 | sacct_def | 4 | const | 60513 | Using index condition; Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+-------------------------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ 3 rows in set (0.00 sec) Adding nodes_alloc index speeds up quries with queries like: sacct -i 2-10000 mysql> EXPLAIN SELECT t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user FROM compy_job_table AS t1 LEFT JOIN compy_assoc_table AS t2 ON t1.id_assoc = t2.id_assoc LEFT JOIN compy_resv_table AS t3 ON t1.id_resv = t3.id_resv WHERE ((t1.nodes_alloc between 2 and 10000)) && ((t1.time_start && ((1434384740 BETWEEN t1.time_start AND t1.time_end) || (t1.time_start BETWEEN 1434384740 AND 1434384741)))) GROUP BY id_job , time_submit DESC; +----+-------------+-------+--------+----------------+---------+---------+-------------------------+--------+----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+----------------+---------+---------+-------------------------+--------+----------------------------------------------+ | 1 | SIMPLE | t1 | ALL | id_job,rollup2 | NULL | NULL | NULL | 117549 | Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | 1411_master.t1.id_assoc | 1 | NULL | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | 1411_master.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+----------------+---------+---------+-------------------------+--------+----------------------------------------------+ 3 rows in set (0.00 sec) mysql> EXPLAIN SELECT t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user FROM compy_job_table AS t1 LEFT JOIN compy_assoc_table AS t2 ON t1.id_assoc = t2.id_assoc LEFT JOIN compy_resv_table AS t3 ON t1.id_resv = t3.id_resv WHERE ((t1.nodes_alloc between 2 and 10000)) && ((t1.time_start && ((1434384740 BETWEEN t1.time_start AND t1.time_end) || (t1.time_start BETWEEN 1434384740 AND 1434384741)))) GROUP BY id_job , time_submit DESC; +----+-------------+-------+--------+----------------------------+-------------+---------+-------------------------+------+---------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+----------------------------+-------------+---------+-------------------------+------+---------------------------------------------------------------------+ | 1 | SIMPLE | t1 | range | id_job,rollup2,nodes_alloc | nodes_alloc | 4 | NULL | 720 | Using index condition; Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | 1411_master.t1.id_assoc | 1 | NULL | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | 1411_master.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+----------------------------+-------------+---------+-------------------------+------+---------------------------------------------------------------------+ 3 rows in set (0.00 sec)
-