- 03 Aug, 2015 3 commits
-
-
Dorian Krause authored
This commit adds support for free_mem as part of the sinfo command using the key 'e' ('f' and 'F' are already taken).
-
Dorian Krause authored
This commit adds information about the free memory to the output created by slurm_sprint_node_table(). The unit of FreeMem is MiB like RealMemory.
-
Dorian Krause authored
This commit adds a new field free_mem to the node_info struct and augments the 15.08 RPC protocol to exchange this information as part of the ping and registration status messages. The free memory is taken from the sysinfo() information on systems that support this function. Overall, the implementation sticks to the way cpu_load (the 5 min load average) is handled.
-
- 31 Jul, 2015 5 commits
-
-
Danny Auble authored
note the change and remove the new time limit if the job is requeued.
-
Danny Auble authored
-
Danny Auble authored
and give different error codes for different limits violated.
-
Morris Jette authored
A job being considered for backfill scheduling can have its time limit altered to the partition limit, job minimum time limit, etc. This fixes two places where the backfill scheduler could leave the job in a pending state with a time limit different from it's original value.
-
Morris Jette authored
add more startup/initialization logic fix some locking issues fix some issues related to persistent buffers
-
- 30 Jul, 2015 4 commits
-
-
Danny Auble authored
-
Morris Jette authored
argument "size" to _print_topo_record() could be zero, resulting in invalid array size specification of zero. Dead assignment to "len" in second call to _print_topo_record().
-
Morris Jette authored
Add accounting for persistent buffer space Fixes for data structure removal
-
David Bigagli authored
-
- 29 Jul, 2015 1 commit
-
-
Morris Jette authored
Add "job_process" function. Minor clean-up from code review.
-
- 28 Jul, 2015 3 commits
-
-
Morris Jette authored
Add logic to purge persistent burst buffers. Create dummy job script as needed for buffer teardown Multiple persistent buffer create/destroy calls in single job working
-
Thomas Cadeau authored
-
Thomas Cadeau authored
-
- 27 Jul, 2015 11 commits
-
-
Morris Jette authored
-
Brian Christiansen authored
Bug 1819 Composite indexes search left to right. E.g. an index of (inx1, inx2, inx3) will search from left to right. inx2 can't be used in a where statement by itself, it requires inx1 to be present (inx3 can be optional). For the rollup index having time_end first speeds up the below query. The actual rollup queries still benefit from the original rollup index. sacct -S 07/22-09:41:36 -E 07/22-09:42:37 -i 1-4 -ojobid,start,end,nnodes,nodelist -n -a: mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where ((t1.nodes_alloc between 1 and 4)) && ((t1.time_eligible < 1437550957 && (t1.time_end >= 1437550896 || t1.time_end = 0))) group by id_job, time_submit desc; +----+-------------+-------+--------+---------------+---------+---------+------------------------+--------+----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+---------------+---------+---------+------------------------+--------+----------------------------------------------+ | 1 | SIMPLE | t1 | ALL | id_job,rollup | NULL | NULL | NULL | 120953 | Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+---------------+---------+---------+------------------------+--------+----------------------------------------------+ 3 rows in set (0.00 sec) mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where ((t1.nodes_alloc between 1 and 4)) && ((t1.time_eligible < 1437550957 && (t1.time_end >= 1437550896 || t1.time_end = 0))) group by id_job, time_submit desc; +----+-------------+-------+--------+-----------------------+---------+---------+------------------------+------+---------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+-----------------------+---------+---------+------------------------+------+---------------------------------------------------------------------+ | 1 | SIMPLE | t1 | range | id_job,rollup,rollup2 | rollup2 | 8 | NULL | 6 | Using index condition; Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+-----------------------+---------+---------+------------------------+------+---------------------------------------------------------------------+ 3 rows in set (0.00 sec) rollup: mysql> explain select job.job_db_inx, job.id_job, job.id_assoc, job.id_wckey, job.array_task_pending, job.time_eligible, job.time_start, job.time_end, job.time_suspended, job.cpus_alloc, job.cpus_req, job.id_resv, SUM(step.consumed_energy) from compy_job_table as job left outer join compy_step_table as step on job.job_db_inx=step.job_db_inx and (step.id_step>=0) where (job.time_eligible < 1420102800 && (job.time_end >= 1420099200 || job.time_end = 0)) group by job.job_db_inx order by job.id_assoc, job.time_eligible; +----+-------------+-------+-------+--------------------------------------------------------------------------------+---------+---------+---------------------------+------+--------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+-------+--------------------------------------------------------------------------------+---------+---------+---------------------------+------+--------------------------------------------------------+ | 1 | SIMPLE | job | range | PRIMARY,id_job,rollup,rollup2,wckey,qos,association,array_job,reserv,sacct_def | rollup | 4 | NULL | 1 | Using index condition; Using temporary; Using filesort | | 1 | SIMPLE | step | ref | PRIMARY | PRIMARY | 4 | slurm_1412.job.job_db_inx | 1 | Using where | +----+-------------+-------+-------+--------------------------------------------------------------------------------+---------+---------+---------------------------+------+--------------------------------------------------------+ 2 rows in set (0.01 sec) A plain sacct is sped up by moving time_end into the middle of the index (ex. id_user, time_end, time_eligible). sacct_def is for sacct calls with a state specified, sacct_def2 is for a plain sacct call. plain sacct: mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where (t1.id_user='1003') && ((t1.time_end >= 1437548400 || t1.time_end = 0)) group by id_job, time_submit desc; +----+-------------+-------+--------+------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ | 1 | SIMPLE | t1 | ref | id_job,sacct_def | sacct_def | 4 | const | 60476 | Using index condition; Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ 3 rows in set (0.00 sec)o mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where (t1.id_user='1003') && ((t1.time_end >= 1437548400 || t1.time_end = 0)) group by id_job, time_submit desc; +----+-------------+-------+--------+-------------------------------------+------------+---------+------------------------+------+--------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+-------------------------------------+------------+---------+------------------------+------+--------------------------------------------------------+ | 1 | SIMPLE | t1 | range | id_job,rollup2,sacct_def,sacct_def2 | sacct_def2 | 8 | NULL | 68 | Using index condition; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+-------------------------------------+------------+---------+------------------------+------+--------------------------------------------------------+ 3 rows in set (0.00 sec) Adding the sacct_def2 index order didn't affect other queries: sacct -s CA,CD,F,R: mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where (t1.id_user='1003') && ((t1.state='4' && (t1.time_end && (t1.time_end >= 1438028802))) || (t1.state='3' && (t1.time_end && (t1.time_end >= 1438028802))) || (t1.state='5' && (t1.time_end && (t1.time_end >= 1438028802))) || (t1.time_start && ((!t1.time_end && t1.state=1) || (1438028802 between t1.time_start and t1.time_end)))) group by id_job, time_submit desc; +----+-------------+-------+--------+-------------------------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+-------------------------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ | 1 | SIMPLE | t1 | ref | id_job,rollup2,sacct_def,sacct_def2 | sacct_def | 4 | const | 60513 | Using index condition; Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+-------------------------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ 3 rows in set (0.00 sec) Adding nodes_alloc index speeds up quries with queries like: sacct -i 2-10000 mysql> EXPLAIN SELECT t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user FROM compy_job_table AS t1 LEFT JOIN compy_assoc_table AS t2 ON t1.id_assoc = t2.id_assoc LEFT JOIN compy_resv_table AS t3 ON t1.id_resv = t3.id_resv WHERE ((t1.nodes_alloc between 2 and 10000)) && ((t1.time_start && ((1434384740 BETWEEN t1.time_start AND t1.time_end) || (t1.time_start BETWEEN 1434384740 AND 1434384741)))) GROUP BY id_job , time_submit DESC; +----+-------------+-------+--------+----------------+---------+---------+-------------------------+--------+----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+----------------+---------+---------+-------------------------+--------+----------------------------------------------+ | 1 | SIMPLE | t1 | ALL | id_job,rollup2 | NULL | NULL | NULL | 117549 | Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | 1411_master.t1.id_assoc | 1 | NULL | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | 1411_master.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+----------------+---------+---------+-------------------------+--------+----------------------------------------------+ 3 rows in set (0.00 sec) mysql> EXPLAIN SELECT t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user FROM compy_job_table AS t1 LEFT JOIN compy_assoc_table AS t2 ON t1.id_assoc = t2.id_assoc LEFT JOIN compy_resv_table AS t3 ON t1.id_resv = t3.id_resv WHERE ((t1.nodes_alloc between 2 and 10000)) && ((t1.time_start && ((1434384740 BETWEEN t1.time_start AND t1.time_end) || (t1.time_start BETWEEN 1434384740 AND 1434384741)))) GROUP BY id_job , time_submit DESC; +----+-------------+-------+--------+----------------------------+-------------+---------+-------------------------+------+---------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+----------------------------+-------------+---------+-------------------------+------+---------------------------------------------------------------------+ | 1 | SIMPLE | t1 | range | id_job,rollup2,nodes_alloc | nodes_alloc | 4 | NULL | 720 | Using index condition; Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | 1411_master.t1.id_assoc | 1 | NULL | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | 1411_master.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+----------------------------+-------------+---------+-------------------------+------+---------------------------------------------------------------------+ 3 rows in set (0.00 sec)
-
Morris Jette authored
No change in functionality
-
Morris Jette authored
If node definitions in slurm.conf are spread across multiple lines and topology/tree is configured, then sub-optimal node selection can occur. bug 1645
-
Nathan Yee authored
This also adds the ability to display allocated memory on a node bug 1804
-
Morris Jette authored
-
Morris Jette authored
Conflicts: slurm/slurm.h.in src/slurmctld/job_scheduler.c src/slurmd/slurmstepd/task.c
-
Dorian Krause authored
-
Dominik Bartkiewicz authored
-
Morris Jette authored
Rather than generating loads of log messages about too much time being used to build the job queue every few seconds, log it only every 10 minutes. bug 1827
-
Morris Jette authored
Log the number of job-partition pairs added to the queue for job scheduling. bug 1827
-
- 24 Jul, 2015 4 commits
-
-
Morris Jette authored
Minor updates to web page document Minor improvements to error message logic
-
Morris Jette authored
Improve logging Disable stage-in/out if only persistent buffers used End-to-end with persistent buffer works Improve user error information
-
Danny Auble authored
-
Morris Jette authored
Also, use first pool as default if none configured Several changes for updated API calls
-
- 23 Jul, 2015 3 commits
-
-
Morris Jette authored
-
Morris Jette authored
-
Morris Jette authored
On Cray we were seeing an srun error reading slurmstepd message header. This was due to a shutdown race condition and the error message has been removed. Cray: Disable LDAP references from slurmstepd on job launch due for improved scalability. Document EioTimeout configuration parameter for large system bug 1786
-
- 22 Jul, 2015 6 commits
-
-
Morris Jette authored
If an RPC is being sent to a node and the destination job no longer exists, log the message "job not running" rather than looking up the error and reporting "invalid job id".
-
Morris Jette authored
-
Morris Jette authored
No change in logic, just renamed some variable and function names for greater clarity
-
Morris Jette authored
-
Nicolas Joly authored
Previously only batch job completions were being captured. bug 1820
-
Morris Jette authored
-