- 28 Jul, 2015 3 commits
-
-
Morris Jette authored
Add logic to purge persistent burst buffers. Create dummy job script as needed for buffer teardown Multiple persistent buffer create/destroy calls in single job working
-
Thomas Cadeau authored
-
Thomas Cadeau authored
-
- 27 Jul, 2015 11 commits
-
-
Morris Jette authored
-
Brian Christiansen authored
Bug 1819 Composite indexes search left to right. E.g. an index of (inx1, inx2, inx3) will search from left to right. inx2 can't be used in a where statement by itself, it requires inx1 to be present (inx3 can be optional). For the rollup index having time_end first speeds up the below query. The actual rollup queries still benefit from the original rollup index. sacct -S 07/22-09:41:36 -E 07/22-09:42:37 -i 1-4 -ojobid,start,end,nnodes,nodelist -n -a: mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where ((t1.nodes_alloc between 1 and 4)) && ((t1.time_eligible < 1437550957 && (t1.time_end >= 1437550896 || t1.time_end = 0))) group by id_job, time_submit desc; +----+-------------+-------+--------+---------------+---------+---------+------------------------+--------+----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+---------------+---------+---------+------------------------+--------+----------------------------------------------+ | 1 | SIMPLE | t1 | ALL | id_job,rollup | NULL | NULL | NULL | 120953 | Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+---------------+---------+---------+------------------------+--------+----------------------------------------------+ 3 rows in set (0.00 sec) mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where ((t1.nodes_alloc between 1 and 4)) && ((t1.time_eligible < 1437550957 && (t1.time_end >= 1437550896 || t1.time_end = 0))) group by id_job, time_submit desc; +----+-------------+-------+--------+-----------------------+---------+---------+------------------------+------+---------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+-----------------------+---------+---------+------------------------+------+---------------------------------------------------------------------+ | 1 | SIMPLE | t1 | range | id_job,rollup,rollup2 | rollup2 | 8 | NULL | 6 | Using index condition; Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+-----------------------+---------+---------+------------------------+------+---------------------------------------------------------------------+ 3 rows in set (0.00 sec) rollup: mysql> explain select job.job_db_inx, job.id_job, job.id_assoc, job.id_wckey, job.array_task_pending, job.time_eligible, job.time_start, job.time_end, job.time_suspended, job.cpus_alloc, job.cpus_req, job.id_resv, SUM(step.consumed_energy) from compy_job_table as job left outer join compy_step_table as step on job.job_db_inx=step.job_db_inx and (step.id_step>=0) where (job.time_eligible < 1420102800 && (job.time_end >= 1420099200 || job.time_end = 0)) group by job.job_db_inx order by job.id_assoc, job.time_eligible; +----+-------------+-------+-------+--------------------------------------------------------------------------------+---------+---------+---------------------------+------+--------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+-------+--------------------------------------------------------------------------------+---------+---------+---------------------------+------+--------------------------------------------------------+ | 1 | SIMPLE | job | range | PRIMARY,id_job,rollup,rollup2,wckey,qos,association,array_job,reserv,sacct_def | rollup | 4 | NULL | 1 | Using index condition; Using temporary; Using filesort | | 1 | SIMPLE | step | ref | PRIMARY | PRIMARY | 4 | slurm_1412.job.job_db_inx | 1 | Using where | +----+-------------+-------+-------+--------------------------------------------------------------------------------+---------+---------+---------------------------+------+--------------------------------------------------------+ 2 rows in set (0.01 sec) A plain sacct is sped up by moving time_end into the middle of the index (ex. id_user, time_end, time_eligible). sacct_def is for sacct calls with a state specified, sacct_def2 is for a plain sacct call. plain sacct: mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where (t1.id_user='1003') && ((t1.time_end >= 1437548400 || t1.time_end = 0)) group by id_job, time_submit desc; +----+-------------+-------+--------+------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ | 1 | SIMPLE | t1 | ref | id_job,sacct_def | sacct_def | 4 | const | 60476 | Using index condition; Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ 3 rows in set (0.00 sec)o mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where (t1.id_user='1003') && ((t1.time_end >= 1437548400 || t1.time_end = 0)) group by id_job, time_submit desc; +----+-------------+-------+--------+-------------------------------------+------------+---------+------------------------+------+--------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+-------------------------------------+------------+---------+------------------------+------+--------------------------------------------------------+ | 1 | SIMPLE | t1 | range | id_job,rollup2,sacct_def,sacct_def2 | sacct_def2 | 8 | NULL | 68 | Using index condition; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+-------------------------------------+------------+---------+------------------------+------+--------------------------------------------------------+ 3 rows in set (0.00 sec) Adding the sacct_def2 index order didn't affect other queries: sacct -s CA,CD,F,R: mysql> explain select t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_alloc, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user from compy_job_table as t1 left join compy_assoc_table as t2 on t1.id_assoc=t2.id_assoc left join compy_resv_table as t3 on t1.id_resv=t3.id_resv where (t1.id_user='1003') && ((t1.state='4' && (t1.time_end && (t1.time_end >= 1438028802))) || (t1.state='3' && (t1.time_end && (t1.time_end >= 1438028802))) || (t1.state='5' && (t1.time_end && (t1.time_end >= 1438028802))) || (t1.time_start && ((!t1.time_end && t1.state=1) || (1438028802 between t1.time_start and t1.time_end)))) group by id_job, time_submit desc; +----+-------------+-------+--------+-------------------------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+-------------------------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ | 1 | SIMPLE | t1 | ref | id_job,rollup2,sacct_def,sacct_def2 | sacct_def | 4 | const | 60513 | Using index condition; Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_assoc | 1 | Using where | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | slurm_1412.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+-------------------------------------+-----------+---------+------------------------+-------+---------------------------------------------------------------------+ 3 rows in set (0.00 sec) Adding nodes_alloc index speeds up quries with queries like: sacct -i 2-10000 mysql> EXPLAIN SELECT t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user FROM compy_job_table AS t1 LEFT JOIN compy_assoc_table AS t2 ON t1.id_assoc = t2.id_assoc LEFT JOIN compy_resv_table AS t3 ON t1.id_resv = t3.id_resv WHERE ((t1.nodes_alloc between 2 and 10000)) && ((t1.time_start && ((1434384740 BETWEEN t1.time_start AND t1.time_end) || (t1.time_start BETWEEN 1434384740 AND 1434384741)))) GROUP BY id_job , time_submit DESC; +----+-------------+-------+--------+----------------+---------+---------+-------------------------+--------+----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+----------------+---------+---------+-------------------------+--------+----------------------------------------------+ | 1 | SIMPLE | t1 | ALL | id_job,rollup2 | NULL | NULL | NULL | 117549 | Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | 1411_master.t1.id_assoc | 1 | NULL | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | 1411_master.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+----------------+---------+---------+-------------------------+--------+----------------------------------------------+ 3 rows in set (0.00 sec) mysql> EXPLAIN SELECT t1.account, t1.array_max_tasks, t1.array_task_str, t1.cpus_req, t1.derived_ec, t1.derived_es, t1.exit_code, t1.id_array_job, t1.id_array_task, t1.id_assoc, t1.id_block, t1.id_group, t1.id_job, t1.id_qos, t1.id_resv, t3.resv_name, t1.id_user, t1.id_wckey, t1.job_db_inx, t1.job_name, t1.kill_requid, t1.mem_req, t1.node_inx, t1.nodelist, t1.nodes_alloc, t1.partition, t1.priority, t1.state, t1.time_eligible, t1.time_end, t1.time_start, t1.time_submit, t1.time_suspended, t1.timelimit, t1.track_steps, t1.wckey, t1.gres_alloc, t1.gres_req, t1.gres_used, t2.acct, t2.lft, t2.user FROM compy_job_table AS t1 LEFT JOIN compy_assoc_table AS t2 ON t1.id_assoc = t2.id_assoc LEFT JOIN compy_resv_table AS t3 ON t1.id_resv = t3.id_resv WHERE ((t1.nodes_alloc between 2 and 10000)) && ((t1.time_start && ((1434384740 BETWEEN t1.time_start AND t1.time_end) || (t1.time_start BETWEEN 1434384740 AND 1434384741)))) GROUP BY id_job , time_submit DESC; +----+-------------+-------+--------+----------------------------+-------------+---------+-------------------------+------+---------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+----------------------------+-------------+---------+-------------------------+------+---------------------------------------------------------------------+ | 1 | SIMPLE | t1 | range | id_job,rollup2,nodes_alloc | nodes_alloc | 4 | NULL | 720 | Using index condition; Using where; Using temporary; Using filesort | | 1 | SIMPLE | t2 | eq_ref | PRIMARY | PRIMARY | 4 | 1411_master.t1.id_assoc | 1 | NULL | | 1 | SIMPLE | t3 | ref | PRIMARY | PRIMARY | 4 | 1411_master.t1.id_resv | 1 | NULL | +----+-------------+-------+--------+----------------------------+-------------+---------+-------------------------+------+---------------------------------------------------------------------+ 3 rows in set (0.00 sec)
-
Morris Jette authored
No change in functionality
-
Morris Jette authored
If node definitions in slurm.conf are spread across multiple lines and topology/tree is configured, then sub-optimal node selection can occur. bug 1645
-
Nathan Yee authored
This also adds the ability to display allocated memory on a node bug 1804
-
Morris Jette authored
-
Morris Jette authored
Conflicts: slurm/slurm.h.in src/slurmctld/job_scheduler.c src/slurmd/slurmstepd/task.c
-
Dorian Krause authored
-
Dominik Bartkiewicz authored
-
Morris Jette authored
Rather than generating loads of log messages about too much time being used to build the job queue every few seconds, log it only every 10 minutes. bug 1827
-
Morris Jette authored
Log the number of job-partition pairs added to the queue for job scheduling. bug 1827
-
- 24 Jul, 2015 4 commits
-
-
Morris Jette authored
Minor updates to web page document Minor improvements to error message logic
-
Morris Jette authored
Improve logging Disable stage-in/out if only persistent buffers used End-to-end with persistent buffer works Improve user error information
-
Danny Auble authored
-
Morris Jette authored
Also, use first pool as default if none configured Several changes for updated API calls
-
- 23 Jul, 2015 3 commits
-
-
Morris Jette authored
-
Morris Jette authored
-
Morris Jette authored
On Cray we were seeing an srun error reading slurmstepd message header. This was due to a shutdown race condition and the error message has been removed. Cray: Disable LDAP references from slurmstepd on job launch due for improved scalability. Document EioTimeout configuration parameter for large system bug 1786
-
- 22 Jul, 2015 10 commits
-
-
Morris Jette authored
If an RPC is being sent to a node and the destination job no longer exists, log the message "job not running" rather than looking up the error and reporting "invalid job id".
-
Morris Jette authored
-
Morris Jette authored
No change in logic, just renamed some variable and function names for greater clarity
-
Morris Jette authored
-
Nicolas Joly authored
Previously only batch job completions were being captured. bug 1820
-
Morris Jette authored
-
Morris Jette authored
-
David Bigagli authored
-
Morris Jette authored
If a job was running on the node when slurmctld restarted. The slurmd would notify slurmctld when the job ended and slurmctld would change its state from UNKNOWN to IDLE, at least if the job termination happened prior to the slurmd being asked for configuration information. The configuration information might then not be collected for some time. I've modified the code to address this problem and try to collect configuration information from every node after slurmctld startup, eliminating this race condition. bug 1805
-
Brian Christiansen authored
Bug 1208
-
- 21 Jul, 2015 8 commits
-
-
Morris Jette authored
-
Morris Jette authored
-
Chandler Wilkerson authored
This patch provides a rewrite of how /proc/cpuinfo is parsed in common_jag.c, as the original code made the incorrect assumption that cpuinfo follows a sane format across architectures ;-) The motivation for this patch is that the original code was producing stack smashing on a POWER7 running RHEL6.4 Red Hat adds -fstack-protector along with a lot of other protective CFLAGS when building RPMs. The code ran okay with -fno-stack-protector, but that is not the best work-around. So, the relevant /proc/cpuinfo line on an Intel (Xeon X5675) system looks like: cpu MHz : 3066.915 Whereas the relevant line in a POWER7 system is clock : 3550.000000MHz My patch replaces the assumption that the relevant number starts 11 characters into the string with another assumption: That the relevant number starts two characters after a colon in a string that matches (M|G)Hz. All in all, the function has a few more calls, which may be a performance issue if it has to be called multiple times, but since the section I edited only gets evaluated if we don't know the cpu frequency, getting it right will actually result in fewer string operations and unnecessary opens of /proc/cpuinfo for systems likewise affected. Finally, I also read the actual value into a double and multiply it up to the size indicated by the suffix, so we end up with KHz? It was unclear what the original code intended, since it matched both MHz and GHz, replaced the decimal point with a zero, and read the result as an int. -- Chandler Wilkerson Center for Research Computing Rice University
-
Danny Auble authored
This reverts commit 2c95e2d2. Conflicts: src/plugins/select/alps/basil_interface.c This is related to bug 1822. It isn't clear why the code was taken out in this commit in the first place and based off of commit 2e2de6a4 (which is the reason for the conflict) we tried unsuccessfully to put it back. It appears the only difference here is the addition of always setting mppnppn = 1 instead of always to job_ptr->details->ntasks_per_node when no ntasks is set. This appears to only be an issue with salloc or sbatch as ntasks is always set for srun.
-
David Gloe authored
I've attached a patch to add a new script and templates that create slurm.conf and gres.conf on a Cray SMW. The script parses the output of xthwinv, combines nodes with the same hardware, and writes the output to slurm.conf. It also uses NodeName in gres.conf to allow having the same gres.conf file on all nodes. I removed the settings in slurm.conf that we had commented out or set to the default.
-
Morris Jette authored
-
Morris Jette authored
-
david authored
-
- 20 Jul, 2015 1 commit
-
-
Morris Jette authored
-