Newer
Older
FastSchedule=0 AND low CPUs count in Slurm's node configuration.
- Add new and undocumented scancel option, --ctld, to route signal
requests through slurmctld rather than directly to slurmd daemons.
Useful for testing purposes.
- Fixed issue with hostfile support not working in a job step.
- Set supplemental groups for SlurmUser in slurmctld daemon, from
Anne Marie Wunderlin, Bull.
- In jobcomp/script: Add ACCOUNT and PROCS (count) to environment
variables set. Fix bug that prevented UID and JOBID from being
overwritten.
* Changes in SLURM 1.1.34
=========================
- Insure that slurm_signal_job_step() is defined in srun for mvapich
and mpichgm error conditions.
- Modify /etc/init.d/slurm restart command to wait for daemon to terminate
before starting a new one
- Permit job steps to be started on draining nodes that have already
been allocated to that job.
- Prevent backup slurmctld from purging pending batch job scripts when a
SIGHUP is received.
- BLUEGENE - check to make sure set_block_user works when the block
is in a ready state.
- Fix to slurmstepd to not use local variables in a pthread create.
- In sched/wiki2 - add wiki.conf paramter HostFormat specifying
format of hostlists exchanged between Slurm and Moab (experimental).
- mpi/mvapich: Support Adam Moody's fast MPI initialization protocol
(MVAPICH protocol version 8).
* Changes in SLURM 1.1.33
=========================
- sched/wiki2 - Do not wait for job completion before permitting
additional jobs to be scheduled.
- Add srun SLURM_EXCLUSIVE environment variable support, from
Gilles Civario (Bull).
- sched/wiki2 - Report job's node sharing options.
- sched/wiki2 - If SchedulerPort is in use, retry opening it indefinitely.
- sched/wiki2 - Add support for changing the size of a pending job.
- BLUEGENE - Fix to correctly look at downed/drained nodes with picking
a block to run a job and not confuse it with another running job.
* Changes in SLURM 1.1.32
=========================
- If a job's stdout/err file names are unusable (bad path), use the
default names.
- sched/wiki2 - Fix logic to be compatible with select/cons_res plugin
for allocating individual processors within nodes.
- Fix job end time calculation when changed from an initial value of
INFINITE.
Christopher J. Morrone
committed
* Changes in SLURM 1.1.31
=========================
- Correctly identify a user's login shell when running "srun -b --uid"
as root. Use the --uid field for the /etc/passwd lookup instead of
getuid().
* Changes in SLURM 1.1.30
=========================
Christopher J. Morrone
committed
- Fix to make sure users don't include and exclude the same node in
their srun line.
- mpi/mvapich: Forcibly terminate job 60s after first MPI_Abort()
to avoid waiting indefinitely for hung processes.
- proctrack/sgi_job: Fix segv when destroying an active job container
with processes still running.
- Abort a job's stdout/err to srun if not processed within 5 minutes
(prevents node hanging in completing state if the srun is stopped).
* Changes in SLURM 1.1.29
=========================
- Fix bug which could leave orphan process put into background from
batch script.
* Changes in SLURM 1.1.28
=========================
- BLUEGENE - Fixed issue with nodes that return to service outside of an
admin state is now updated in the bluegene plugin.
- Fix for --get-user-env parsing of non-printing characters in users' logins.
- Restore "squeue -n localhost" support.
- Report lack of PATH env var as verbose message, not error in srun.
* Changes in SLURM 1.1.27
=========================
- Fix possible race condition for two simultaneous "scontrol show config"
calls resulting in slurm_xfree() Error: from read_config.c:642
- BLUEGENE - Put back logic to make a block fail a boot 3 times before
cancelling a users job.
- Fix problem using srun --exclude option for a job step.
- Fix problem generating slurmd error "Unrecognized request: 0" with
some compilers.
* Changes in SLURM 1.1.26
=========================
- In sched/wiki2, fixes for support of job features.
- In sched/wiki2, add "FLAGS=INTERACTIVE;" to GETJOBS response for
non-batch (not srun --batch) jobs.
* Changes in SLURM 1.1.25
=========================
- switch/elan: Fix for "Failed to initialise stats structure" from
libelan when ELAN_STATKEY > MAX_INT.
- Tune PMI support logic for better scalability and performance.
- Fix for running a task on each node of an allocation if not specified.
- In sched/wiki2, set TASKLIST for running jobs.
- In sched/wiki2, set STARTDATE for pending jobs with deferred start.
- Added srun --get-user-env option (for Moab scheduler).
* Changes in SLURM 1.1.24
=========================
- In sched/wiki2, add support for direct "srun --dependency=" use.
- mpi/mvapich: Add support for MVAPICH protocol version 6.
- In sched/wiki2, change "JOBMODIFY" command to "MODIFYJOB".
- In sched/wiki2, change "JOBREQUEUE" command to "REQUEUEJOB".
- For sched/wiki2, permit normal user to specify arbitrary job id.
- In sched/wiki2, set buffer pointer to NULL after free() to avoid
possible memory corruption.
- In sched/wiki2, report a job's exit code on completion.
- For AIX, fix mail for job event notification.
- Add documentation for propagation options in man srun and slurm.conf.
* Changes in SLURM 1.1.23
=========================
- Fix bug in non-blocking connect() code affecting AIX.
Christopher J. Morrone
committed
* Changes in SLURM 1.1.22
=========================
- Add squeue option to print a job step's task count (-o %A).
- Initialize forward_struct to avoid trying to free a bad pointer,
patch from Anton Blanchard (SAMBA).
- In sched/wiki2, fix fatal race condition on slurmctld startup.
- Fix for displaying launching verbose messages for each node under the
tree instead of just the head one.
- Fix job suspend bug, job accounting plugin would SEGV when given a
bad job ID.
* Changes in SLURM 1.1.21
=========================
- BLUEGENE - Wait on a fini to make sure all threads are finished before
cleaning up.
- BLUEGENE - replacements to not destroy lists but just empty it to avoid
losing the pointer to the list in the block allocator.
- BLUEGENE - added --enable-bluegene-emulation configure option to 1.1
- In sched/wiki2, enclose a job's COMMENT value in double quotes.
- In sched/wiki2, support newly defined SIGNALJOB command.
- In sched/wiki2, maintain open event socket, don't open and close
for each event.
- In sched/wiki2, fix for scalability problem starting large jobs.
- Fix logic to execute a batch job step (under an existing resource
allocation) as needed by LSF.
- Patches from Hongjia Cao (pmi finialize issues and type declaration)
- Delete pending job if it's associated partition is deleted.
- fix for handling batch steps completing correctly and setting the
return code.
- Altered ncurses check to make sure programs can link before saying we
have a working curses lib and header.
- Fixed an init issue with forward_struct_init not being set correctly in
a few locations in the slurmd.
- Fix for user to use the NodeHostname (when specified in the slurm.conf file)
to start jobs on.
Christopher J. Morrone
committed
* Changes in SLURM 1.1.20
=========================
- Added new SPANK plugin hook slurm_spank_local_user_init() called
from srun after node allocation.
- Fixed bug with hostfile support not working on a direct srun
* Changes in SLURM 1.1.19
=========================
- BLUEGENE - make sure the order of blocks read in from the bluegene.conf
are created in that order (static mode).
- Fix logic in connect(), slurmctld fail-over was broken in v1.1.18.
- Fix logic to calculate the correct timeout for fan out.
* Changes in SLURM 1.1.18
=========================
- In sched/wiki2, add support for EHost and EHostBackup configuration
parameters in wiki.conf file
- In sched/wiki2, fix memory management bug for JOBWILLRUN command.
- In sched/wiki2, consider job Busy while in Completing state for
KillWait+10 seconds (used to be 30 seconds).
- BLUEGENE - Fixes to allow full block creation on the system and not to add
passthrough nodes to the allocation when creating a block.
- BLUEGENE - Fix deadlock issue with starting and failing jobs at the same
time
- Make connect() non-blocking and poll() with timeout to avoid huge
waits under some conditions.
- Set "ENVIRONMENT=BATCH" environment variable for "srun --batch" jobs only.
- Add logic to save/restore select/cons_res state information.
Christopher J. Morrone
committed
- BLUEGENE - make all sprintf's into snprintf's
Christopher J. Morrone
committed
- Fix for "srun -A" segfault on a node failure.
* Changes in SLURM 1.1.17
=========================
- BLUEGENE - fix to make dynamic partitioning not go create block where
there are nodes that are down or draining.
- Fix srun's default node count with an existing allocation when neither
SLURM_NNODES nor -N are set.
- Stop srun from setting SLURM_DISTRIBUTION under job steps when a
specific was not explicitly requested by the user.
* Changes in SLURM 1.1.16
=========================
- BLUEGENE - fix to make prolog run 5 minutes longer to make sure we have
enough time to free the overlapping blocks when starting a new job on a
block.
- BLUEGENE - edit to the libsched_if.so to read env and look at
MPIRUN_PARTITION to see if we are in slurm or running mpirun natively.
- Plugins are now dlopened RTLD_LAZY instead of RTLD_NOW.
* Changes in SLURM 1.1.15
=========================
- BLUEGENE - fix to be able to create static partitions
- Fixed fanout timeout logic.
- Fix for slurmctld timeout on outgoing message (Hongjia Cao, NUDT.edu.cn).
* Changes in SLURM 1.1.14
=========================
- In sched/wiki2: report job/node id and state only if no changes since
time specified in request.
- In sched/wiki2: include a job's exit code in job state information.
- In sched/wiki2: add event notification logic on job submit and completion.
- In sched/wiki2: add support for JOBWILLRUN command type.
- In sched/wiki2: for job info, include required HOSTLIST if applicable.
- In sched/wiki2: for job info, replace PARTITIONMASK with RCLASS (report
partition name associated with a job, but no task count)
- In sched/wiki2: for job and node info, report all data if TS==0,
volitile data if TS<=update_time, state only if TS>update_time
- In sched/wiki2: add support for CMD=JOBSIGNAL ARG=jobid SIGNAL=name or #
- In sched/wiki2: add support for CMD=JOBMODIFY ARG=jobid [BANK=name]
[TIMELIMIT=minutes] [PARTITION=name]
- In sched/wiki2: add support for CMD=INITIALIZE ARG=[USEHOSTEXP=T|F]
[EPORT=#]; RESPONSE=EPORT=# USEHOSTEXP=T
- In sched/wiki2: fix memory leak.
- Fix sinfo node state filtering when asking for idle nodes that are also
draining.
- Add Fortran extension to slurm_get_rem_time() API.
- Fix bug when changing the time limit of a running job that has previously
been suspended (formerly failed to account for suspend time in setting
termination time).
- fix for step allocation to be able to specify only a few nodes in a
step and ask for more that specified.
- patch from Hongjia Cao for forwarding logic
- BLUEGENE - able to allocate specific nodes without locking up.
- BLUEGENE - better tracking of blocks that are created dynamically,
less hitting the db2.
* Changes in SLURM 1.1.13
=========================
- Fix hang in sched/wiki2 if Moab stops responding responding when
response is outgoing.
- BLUEGENE - fix to make sure the block is good to go when picking it
- BLUEGENE - add libsched_if.so so mpirun doesn't try to create a block
by itself.
- Enable specification of srun --jobid=# option with --batch (for user root).
- Verify that job actually starts when requested by sched/wiki2.
- Add new wiki.conf parameters: EPort and JobAggregationTime for event
notification logic (see wiki.conf man page for details)
* Changes in SLURM 1.1.12
=========================
- Sched/wiki2 to report a job's account as COMMENT response to GETJOBS
request.
- Add srun option "--comment" (maps to job account until slurm v1.2,
needed for Moab scheduler functionality).
- fixed some timeout issues in the controller hopefully stopping all the
issues with excessive timeouts.
- unit conversion (i.e. 1024 => 1k) only happens on bgl systems for node
count.
- Sched/wiki2 to report a job's COMPETETIME and SUSPENDTIME in GETJOBS
response.
- Added support for Mellanox's version of mvapich-0.9.7.
* Changes in SLURM 1.1.11
=========================
- Update file headers adding permission to link with OpenSSL.
- Enable sched/wiki2 message authentication.
- Fix libpmi compilation issue.
- Remove "gcc-c++ python" from slurm.spec BuildRequires. It breaks
the AIX build, so we'll have to find another way to deal with that.
* Changes in SLURM 1.1.10
=========================
-- task distribution fix for steps that are smaller than job allocation.
-- BLUEGENE - fix to only send a success when block was created when trying
to allocate the block.
-- fix so if slurm_send_recv_node_msg fails on the send the auth_cred returned
by the resp is NULL.
-- Fix switch/federation plugin so backup controller can assume control
repeatedly without leaking or corrupting memory.
-- Add new error code (for Maui/Moab scheduler): ESLURM_JOB_HELD
-- Tweak slurmctld's node ping logic to better handle failed nodes with
hierarchical communications fail-over logic.
-- Add support for sched/wiki specific configuration file "wiki.conf".
-- Added sched/wiki2 plugin (new experimental wiki plugin).
* Changes in SLURM 1.1.9
========================
Christopher J. Morrone
committed
-- BLUEGENE - fix to handle a NO_VAL sent in as num procs in the job
description.
-- Fix bug in slurmstepd code for parsing --multi-prog command script.
Parser was failing for commands with no arguments.
-- Fix bug to check unsigned ints correctly in bitstring.c
-- Alter node count covert to kilo to only convert number divisible by
1024 or 512
* Changes in SLURM 1.1.8
========================
-- Added bug fixes (fault-tolerance and memory leaks) from Hongjia Cao
<hjcao@nudt.edu.cn>
-- Gixed some potential BLUEGENE issues with the bridge log file not having
a mutex around the fclose and fopen.
-- BLUEGENE - srun -n procs now regristers correctly
-- Fixed problem with reattach double allocating step_layout->tids
-- BLUEGENE - fix race condition where job is finished before it starts.
* Changes in SLURM 1.1.7
========================
-- BLUEGENE - fixed issue with doing an allocation for nodes since asking
for 32,128, or 512 all mean 1 to the controller.
Christopher J. Morrone
committed
-- Add "Include" directive to slurm.conf files. If "Include" is found
at the beginning of a line followed by whitespace and then
the full path to a file, that file is included inline with the current
slurm.conf file.
* Changes in SLURM 1.1.6
========================
-- Improved task layout for relative positions
-- Fixed heterogeous cpu overcommit issue
-- Fix bug where srun would hang if it ran on one node and that
node's slurmd died
-- Fix bug where srun task layout would be bad when min-max node range is
specified (e.g. "srun -N1-4 ...")
-- Made slurmctld_conf.node_prefix only be set on Bluegene systems.
-- Fixed a race condition in the controller to make it so a plugin thread
wouldn't be able to access the slurmctld_conf structure before it was
filled.
* Changes in SLURM 1.1.5
========================
-- Ignore partition's MaxNodes for SlurmUser and root.
-- Fix possible memory corruption with use of PMI_KVS_Create call.
-- Fix race condition when multiple PMI_KVS_Barrier calls.
-- Fix logic in which slurmctld outgoing RPC requests could get delayed.
-- Fix logic for laying out steps without a hostlist.
* Changes in SLURM 1.1.4
========================
-- Improve error handling in hierarchical communications logic.
* Changes in SLURM 1.1.3
========================
-- Fix big-endian bug in the bitstring code which plagued AIX.
-- Fix bug in handling srun's --multi-prog option, could go off end of buffer.
-- Added support for job step completion (and switch window release) on
subset of allocated nodes.
-- BLUEGENE - removed configure option --with-bg-link bridge is linked with
dlopen now no longer needing fake database so files on frontend node.
-- BLUEGENE - implemented use of rm_get_partition_info instead of
...partitions_info which has made a much better design improving stability.
-- Streamline PMI communications and increase timeouts for highly parallel
jobs. Improves scalability of PMI.
* Changes in SLURM 1.1.2
========================
-- Fix bug in jobcomp/filetxt plugin to report proper NodeCnt when a job
fails due to a node failure.
-- Fix Bluegene configure to work with the new 64bit libs.
-- Fix bug in controller that causes it to segfault when hit with a malformed
message.
-- For "srun --attach=X" to other users job, report an error and exit (it
previously just hung).
-- BLUEGENE - fix for doing correct small block logic on user error.
-- BLUEGENE - Added support in slurmd to create a fake libdb2.so if it
doesn't exist so smap won't seg fault
-- BLUEGENE - "scontrol show job" reports "MaxProcs=None" and "Start=None"
if values are not specified at job submit time
-- Add retry logic for PMI communications, may be needed for highly parallel
jobs.
-- Fix bug in slurmd where variable is used in logging message after freed
(slurmstepd rank info).
-- Fix bug in scontrol show daemons if NodeName=localhost will work now to
display slurmd as place where it is running.
-- Patch from HP for init nodes before init_bitmaps
-- ctrl-c killed sruns will result in job state as cancelled instead of
completed.
-- BLUEGENE - added configure option --with-bg-link to choose dynamic linking
or static linking with the bridgeapi.
* Changes in SLURM 1.1.1
========================
-- Fix bug in packing job suspend/resume RPC.
-- If a user breaks out of srun before the allocation takes place, mark the
job as CANCELLED rather than COMPLETED and change its start and end time
to that time.
-- Fix bug in PMI support that prevented use of second PMI_Barrier call.
This fix is needed for MVAPICH2 use.
-- Add "-V" options to slurmctld and slurmd to print version number and exit.
-- Fix scalability bug in sbcast.
-- Fix bug in cons_res allocation strategy.
-- Fix bug in forwarding with mpi
-- Fix bug sacct forwarding with stat option
-- Added nodeid to sacct stat information
-- cleaned up way slurm_send_recv_node_msg works no more clearing errno
-- Fix error handling bug in the networking code that causes the slurmd to
xassert if the server is not running when the slurmd tries to register.
* Changes in SLURM 1.1.0
========================
-- Fix bug that could temporarily make nodes DOWN when they are really
responding.
-- Fix bug preventing backup slurmctld from responding to PING RPCs.
-- Set "CFLAGS=-DISO8601" before configuration to get ISO8601 format
times for all SLURM commands. NOTE: This may break Moab, Maui, and/or
LSF schedulers.
-- Fix for srun -n and -O options when paired with -b.
Danny Auble
committed
-- Added logic for fanout to failover to forward list if main node is
unreachable
-- sacct also now keeps track of submitted, started and ending times of jobs
-- reinit config file mutex at beginning of slurmstepd to avoid fork issues
Danny Auble
committed
* Changes in SLURM 1.1.0-pre8
=============================
-- Fix bug in enforcement of partition's MaxNodes limit.
Danny Auble
committed
-- BLUEGENE - added support for srun -w option also fixed the geometry option
for srun.
Danny Auble
committed
* Changes in SLURM 1.1.0-pre7
=============================
-- Accounting works for aix systems, use jobacct/aix
-- Support large (over 2GB) files on 32-bit linux systems
Danny Auble
committed
-- changed all writes to safe_write in srun
-- added $float to globals.example in the testsuite
-- Set job's num_proc correctly for jobs that do not have exclusive use
of it's allocated nodes.
-- Change in support for test suite: 'testsuite/expect/globals.example'
is now 'testsuite/expect/globals' and you can override variable
settings with a new file 'testsuite/expect/globals.local'.
-- Job suspend now sends SIGTSTP, sleep(1), sends SIGSTOP for better
MPI support.
-- Bluegene - before assigning a job to a block the plugin will check the bps
to make sure they aren't in error state.
-- Change time format in job completion logging (JobCompType=jobcomp/filetxt)
from "MM/DD HH:MM:SS" to "YYYY-MM-DDTHH:MM:SS", conforming with the ISO8601
standard format.
* Changes in SLURM 1.1.0-pre6
=============================
-- Added logic to "stat" a running job with sacct option -S use -j to specify
job.step
Danny Auble
committed
-- removed jobacct/bluegene (no real need for this) meaning, I don't think
there is a way to gather the data yet.
-- Added support for mapping "%h" in configured SlurmdLog to the hostname.
-- Add PropagatePrioProcess to control propagation of a user's nice value
to spawned tasks (based upon work by Daniel Christians, HP).
Danny Auble
committed
* Changes in SLURM 1.1.0-pre5
=============================
-- Added step completion RPC logic
-- Vastly changed sacct and the jobacct plugin. Read documentation for full
details.
-- Added jobacct plugin for AIX and BlueGene, they currently don't work,
but infrastructure is in place.
-- Add support for srun option --ctrl-comm-ifhn to set PMI communications
address (Hongjia Cao, National University of Defense Technology).
-- Moved safe_read/write to slurm_protocol_defs.h removing multiple copies.
-- Remove vestigial functions slurm_allocate_resources_and_run() and
slurm_free_resource_allocation_and_run_response_msg().
-- Added support for different executable files and arguments by task based
upon a configuration file. See srun's --multi-prog option (based upon
work by Hongjia Cao, National University of Defense Technology).
-- moved the way forward logic waited for fanout logic mostly eliminating
problems with scalability issues.
-- changed -l option in sacct to display different params see sacct/sacct.h
for details.
Danny Auble
committed
* Changes in SLURM 1.1.0-pre4
=============================
-- Bluegene specific - Added support to set bluegene block state to
free/error via scontrol update BlockName
-- Add needed symbol to select/bluegene in order to load plugin.
Danny Auble
committed
* Changes in SLURM 1.1.0-pre3
=============================
-- Added framework for XCPU job launch support.
Christopher J. Morrone
committed
-- New general configuration file parser and slurm.conf handling code.
Allows long lines to be continued on the next line by ending with a "\".
Whitespace is allowed between the key and "=", and between the "=" and
value.
WARNING: A NodeName may now occur only once in a slurm.conf file.
If you want to temporarily make nodes DOWN in the slurm.conf,
use the new DownNodes keyword (see "man slurm.conf").
-- Gracefully handle request to submit batch job from within an existing
batch job.
Moe Jette
committed
-- Warn user attempting to create a job allocation from within an existing job
allocation.
-- Add web page description for proctrack plugin.
-- Add new function slurm_get_rem_time() for job's time limit.
-- JobAcct plugin renamed from "log" to "linux" in preparation for support of
new system types.
WARNING: "JobAcctType=jobacct/log" is no longer supported.
-- Removed vestigal 'bg' names from bluegene plugin and smap
-- InactiveLimit parameter is not enforced for RootOnly partitions.
-- Update select/cons_res web page (Susanne Balle, HP,
cons_res_doc_patch_3_29_06).
-- Build a "slurmd.test" along with slurmd. slurmd.test has the path to
slurmstepd set allowing it to run unmodified out of the builddir for
testing (Mark Grondona).
* Changes in SLURM 1.1.0-pre2
=============================
-- Added "bcast" command to transmit copies of a file to compute nodes
with message fanout.
-- Bluegene specific - Added support for overlapping partitions and
dynamic partitioning.
-- Bluegene specific - Added support for nodecard sized blocks.
-- Added logic to accept 1k for 1024 and so on for --nodes option of srun.
This logic is through display tools such as smap, sinfo, scontrol, and
squeue.
-- Added bluegene.conf man page.
-- Added support for memory affinity, see srun --mem_bind option.
* Changes in SLURM 1.1.0-pre1
=============================
-- New --enable-multiple-slurmd configure parameter to allow running
more than one copy of slurmd on a node at the same time. Only
really useful for developers.
-- New communication is now branched on all processes to slurmd's from
slurmctld and srun launch command. This is done with a tree type
algorithm. Spawn and batch mode work the same as before. New slurm.conf
variable TreeWidth=50 is default. This is the number of threads per
-- Configuration parameter HeartBeatInterval is depracated. Now used half
of SlurmdTimeout and SlurmctldTimeout for communications to slurmd and
slurmctld daemons repsectively.
-- Add hash tables for select/cons_res plugin (Susanne Balle, HP,
patch_02222006).
-- Remove some use of cr_enabled flag in slurmctld job record, use
new flag "test_only" in select_g_job_test() instead.
Christopher J. Morrone
committed
* Changes in SLURM 1.0.17
=========================
-- Set correct user groups for task epilogs.
-- Add more debugging for tracking slow slurmd job initiations
(slurm.hp.replaydebug.patch).
Christopher J. Morrone
committed
* Changes in SLURM 1.0.16
=========================
-- For "srun --attach=X" to other users job, report an error and exit (it
previously just hung).
-- Make sure that "scancel -s KILL" terminates the job just like "scancel"
including deletion of all job steps (Chris Holmes, HP, slurm,patch).
-- Recognize ISO-8859 input to srun as a script (for non-English scripts).
-- switch/elan: Fix bug in propagation of ELAN_STATKEY environment variable.
Christopher J. Morrone
committed
-- Fix bug in slurmstepd IO code that can result in it spinning if a
certain error occurs.
Christopher J. Morrone
committed
-- Remove nodes from srun's required node list if their count exceeds
the number of requested tasks.
-- sched/backfill to schedule around jobs that are hung in a completing
state.
-- Avoid possibly re-running the epilog for a job on slurmctld restart or
reconfig by saving and restoring a hostlist of nodes still completing
the job.
* Changes in SLURM 1.0.15
=========================
-- In srun, reset stdin to blocking mode (if it was originally blocking before
we set it to O_NONBLOCK) on exit to avoid trouble with things like running
srun under a bash shell in an emacs *shell* buffer.
-- Fix srun race condition that occasionally causes segfaults at shutdown
-- Fix obscure locking issues in log.c code.
-- Explicitly close IO related sockets. If an srun gets "stuck", possibly
because of unkillable tasks in its job step, it will not hold many TCP
sockets in the CLOSE_WAIT state.
-- Increase the SLURM protocol timeout from 5 seconds to 10 seconds.
(In 1.2 there will be a slurm.conf parameter for this, rather than having
it hardcoded.)
* Changes in SLURM 1.0.14
=========================
-- Fix for bad xfree() call in auth/munge which can raise an assert().
-- Fix installed fork handlers for the conf mutex for slurmd and slurmstepd.
* Changes in SLURM 1.0.13
=========================
-- Fix for AllowGroups option to work when the /etc/group file doesn't
contain all users in group by adding the uids of the names in /etc/passwd
that have a gid of that which we are looking for.
-- Fix bug in InactiveLimit support that can potentially purge active jobs.
NOTE: This is highly unlikely except on very large AIX clusters.
-- Fix bug for reiniting the config_lock around the control_file in
slurm_protocol_api.c logic has changed in 1.1 so no need to merge
* Changes in SLURM 1.0.12
=========================
-- Report node state of DRAIN rather than DOWN if DOWN with DRAIN flag set.
-- Initialize job->mail_type to 0 (NONE) for job submission.
-- Fix for stalled task stdout/stderr when buffered I/O is used, and
a single line exceeds 4096 bytes.
-- Memory leak fixes for maui plugin (hjcao@nudt.edu.cn)
Christopher J. Morrone
committed
-- Fix for spinning srun when the terminal to which srun is talking
goes away.
-- Don't set avail_node_bitmap for DRAINED nodes on slurmctld reconfig
(can schedule a job on drained node after reconfig).
Christopher J. Morrone
committed
* Changes in SLURM 1.0.11
=========================
-- Fix for slurmstepd hang when launching a task. (Needed to install
list library's atfork handlers).
-- Fix memory leak on AIX (and possibly other architectures) due to
missing pthread_attr_destroy() calls.
-- Fix rare task standard I/O setup bug. When the bug hit, stdin, stdout,
Christopher J. Morrone
committed
or stderr could be an invalid file descriptor.
-- General slurmstepd file descriptor cleanup.
-- Fix memory leak in job accounting logic (Andy Riebs, HP, memory_leak.patch).
Christopher J. Morrone
committed
* Changes in SLURM 1.0.10
=========================
-- Fix for job accounting logic submitted from Andy Riebs to handle issues
with suspending jobs and such. patch file named requeue.patch
-- Make select/cons_res interoperate with mpi/lam plugin for task counts.
-- Fix race condition where srun could seg-fault due to use of logging functions
within pthread after calling log_fini.
-- Code changes for clean build with gcc 2.96 (gcc_2_96.patch, Takao Hatazaki, HP).
-- Add CacheGroups configuration support in configurator.html (configurator.patch,
Takao Hatazaki, HP).
-- Fix bug preventing use of mpich-gm plugin (mpichgm.patch, Takao Hatazaki, HP).
* Changes in SLURM 1.0.9
========================
-- Fix job accounting logic to open new log file on slurmctld reconfig.
(Andy Riebs, slurm.hp.logfile.patch).
-- Fix bug which allows a user to run a batch script on a node not allocated
by the slurmctld.
-- Fix poe MP_HOSTFILE handling bug on AIX.
* Changes in SLURM 1.0.8
========================
-- Fix to communication between slurmd and slurmstepd to allow for partial
reads and writes on their communication pipes.
* Changes in SLURM 1.0.7
========================
-- Change in how AuthType=auth/dummy is handled for security testing.
-- Fix for bluegene systems to allow full system partitions to stay booted
when other jobs are submitted to the queue.
* Changes in SLURM 1.0.6
========================
-- Prevent slurmstepd from crashing when srun attaches to batch job.
* Changes in SLURM 1.0.5
========================
-- Restructure logic for scheduling BlueGene small block jobs. Added
"test_only" flag to select_p_job_test() in select plugin.
-- Correct squeue "NODELIST" output for BlueGene small block jobs.
-- Fix possible deadlock situations on BlueGene plugin on errors.
* Changes in SLURM 1.0.4
========================
-- Release job allocation if step creation fails (especially for BlueGene).
-- Fix bug select/bluegene warm start with changed bglblock layout.
-- Fix bug for queuing full-system BlueGene jobs.
* Changes in SLURM 1.0.3
========================
-- Fix bug that could refuse to queue batch jobs for BlueGene system.
-- Add BlueGene plugin mutex lock for reconfig.
-- Ignore BlueGene bgljobs in ERROR state (don't try to kill).
-- Fix job accounting for batch jobs (Andy Riebs, HP,
slurm.hp.jobacct_divby0a.patch).
-- Added proctrack/linuxproc.so to the main RPM.
-- Added mutex around bridge api file to avoid locking up the api.
-- BlueGene mod: Terminate slurm_prolog and slurm_epilog immediately if
SLURM_JOBID environment variable is invalid.
-- Federation driver: allow selection of a sepecific switch interface
(sni0, sni1, etc.) with -euidevice/MP_EUIDEVICE.
-- Return an error for "scontrol reconfig" if there is already one in
progress
* Changes in SLURM 1.0.2
========================
-- Correctly report DRAINED node state as type OTHER for "sinfo --summarize".
-- Fixes in sacct use of malloc (Andy Riebs, HP, sacct_malloc.patch).
-- Smap mods: eliminate screen flicker, fix window resize, report more clear
message if window too small (Dan Palermo, HP, patch.1.0.0.1.060126.smap).
-- Sacct mods for inconsistent records (race condition) and replace --debug
option with --verbose (Andy Riebs, HP, slurm.hp.sacct_exp_vvv.patch).
-- scancel of a job step will now send a job-step-completed message
to the controller after verifying that the step has completed on all nodes.
-- Fix task layout bug in srun.
-- Added times to node "Reason" field when set down for insufficient
resources or if not responding.
-- Validate operation with Elan switch and heterogeneous nodes.
* Changes in SLURM 1.0.1
========================
-- Assorted updates and clarifications in documentation.
-- Detect which munge installation to use 32/64 bit.
-- Fix sinfo filtering bug, especially "sinfo -R" output.
-- Fix node state change bug, resuming down or drained nodes.
-- Fix "scontrol show config" to display JobCredentialPrivateKey instead
of JobCredPrivateKey and JobCredentialPublicCertificate instead of
JobCredPublicKey. They now match the options in the slurm.conf.
-- Fix bug in job accounting for very long node list records (Andy Riebs,
HP, sacct_buf.patch).
-- BLUEGENE SPECIFIC - added load function to smap to load an already
exsistant bluegene.conf file.
-- Fix bug in sacct: If user requests specific job or job step ID,
only the last one with that ID will be reported. If multiple
nodes fail, the job has its state recorded as "JOB_TERMINATED...nf"
(Andy Riebs, HP, slurm.hp.sacct_dup.patch).
-- Fix some inconsistencies in sacct's help message (Andy Riebs, HP,
slurm.hp.sacct_help.patch).
-- Validate input to sacct command and allows embedded spaces in
arguments (Andy Riebs, HP, slurm.hp.sacct_validate.patch).
* Changes in SLURM 0.7.0-pre8
=============================
-- BGL specific -- bug fix for smap configure function down configuration
Christopher J. Morrone
committed
-- Add slurmd cache for group IDs (Takao Hatazaki, HP).
-- Fix bug in processing of "#SLURM" batch script option parsing.
* Changes in SLURM 0.7.0-pre7
=============================
-- Fix issue with NODE_STATE_COMPLETING, could start job on node before
epilog completed.
-- Added some infrastructure for job suspend/resume (scontrol, api, and
slurmctld stub).
-- Set job's num_procs to the actual processor count allocated to the job.
-- Fix bug in HAVE_FRONT_END support for cluster emulation.
* Changes in SLURM 0.7.0-pre6
=============================
-- Added support for task affinity for binding tasks to CPUs (Daniel
Palermo, HP).
-- Integrate task affinity support with configuration, add validation
test.
* Changes in SLURM 0.7.0-pre5
=============================
-- Enhanced performance and debugging for slurmctld reconfiguration.
-- Add "scontrol update Jobid=# Nice=#" support.
-- Basic slurmctld and tool functionality validated to 16k nodes.
-- squeue and smap now display correct info for jobs in bluegene enviornment.
-- Fix setting of SLURM_NODELIST for batch jobs.
-- Add SubmitTime to job information available for display.
-- API function slurm_confirm_allocation() has been marked OBSOLETE
and will go away in some future version of SLURM. Use
-- New API calls slurm_signal_job and slurm_signal_job_step to send
signals directly to the slurmds without triggering the shutdown sequence.
-- remove "uid" from old_job_alloc_msg_t, no longer needed.
-- Several bug fixes in maui scheduler plugin from Dave Jackon
(Cluster Resources).
* Changes in SLURM 0.7.0-pre4
=============================
-- Remove BNR libary functions and add those for PMI (KVS and basic
MPI-1 functions only for now)
-- Added Hostfile support for POE and srun. MP_HOSTFILE env var to set
location of hostfile. Tasks will run from list order in the file.
-- Removes the slurmd's use of SysV shared memory. Instead the slurmd
communicates with the slurmstepd processes through the slurmstepd's
new named unix domain socket. The "stepd_api" is used to talk to the
slurmstepd (src/slurmd/common/stepd_api.[ch]).
-- Bluegene specific - bluegene block allocator will find most any
partition size now. Added support to start at any point in smap
to request a partition instead of always starting at 000.
-- Bluegene specific - Support to smap to down or bring up nodes in
configure mode. Added commands include allup, alldown,
up [range], down [range]
-- Time format in sinfo/squeue/smap/sacct changed from D:HH:MM:SS to
D-HH:MM:SS per POSIX standards document.
-- Treat scontrol update request without any requested changes as an
error condition.
-- Bluegene plugin renamed with BG instead of BGL. partition_allocator moved
into bluegene plugin and renamed block_allocator. Format for bluegene.conf
file changed also. Read bluegene html page. Code is backwards compatable
smap will generate in new form
-- Add srun option --nice to give user some control over job priority.
* Changes in SLURM 0.7.0-pre3
=============================
-- Restructure node states: DRAINING and DRAINED states are replaced
with a DRAIN flag. COMPLETING state is changed to a COMPLETING flag.
-- Test suite moved into testsuite/expect from separate repository.
-- Added new document describing slurm APIs (doc/html/api.html).
-- Permit nodes to be in multiple partitions simultaneously.
* Changes in SLURM 0.7.0-pre2
=============================
-- New stdio protocol. Now srun has just a single TCP stream to each node
of a job-step. srun and slurmd comminicate over the TCP stream using a
simple messaging protocol.
-- Added task plugin and use task prolog/epilog(s).
-- New slurmd_step functionality added. Fork exec instead of using shared
memory. Not completely tested.
-- BGL small partition logic in place in plugin and smap. Scheduler needs
to be rewritten to handle multiple partitions on a single node. No
documentation written on process yet.
-- If running select/bluegene plugin without access to BGL DB2, then
full-system bglblock is of system size defined in bluegene.conf.
* Changes in SLURM 0.7.0-pre1
=============================
-- Support defered initiation of job (e.g. srun --begin=11:30 ...).
-- Add support for srun --cpus-per-task through task allocation in
slurmctld.
-- fixed partition_allocator to work without curses
-- made change to srun to start message thread before other threads
to make sure localtime doesn't interfere.
-- Added new RPCs for slurmctld REQUEST_TERMINATE_JOB or TASKS,
REQUEST_KILL_JOB/TASKS changed to REQUEST_SIGNAL_JOB/TASKS.
-- Add support for e-mail notification on job state changes.
-- Some infrastructure added for task launch controls (slurm.conf:
TaskProlog, TaskEpilog, TaskPlugin; srun --task-prolog, --task-epilog).
* Changes in SLURM 0.6.11
=========================
-- Fix bug in sinfo partition sorting order.
-- Fix bugs in srun use of #SLURM options in batch script.
-- Use full Elan credential space rather than re-using credentials as soon
as job step completes (helps with fault-tolerance).
* Changes in SLURM 0.6.10
=========================
-- Fix for slurmd job termination logic (could hang in COMPLETING state).
-- Sacct bug fixes: Report correct user name for job step, show "uid.gid"
as fifth field of job step record (Andy Riebs, slurm.hp.sacct_uid.patch).
-- Add job_id to maui scheduler plugin start job status message.
-- Fix for srun's handling of null characters in stdout or stderr.
-- Update job accounting for larger systems (Andy Riebs, uptodate.patch).
Christopher J. Morrone
committed
-- Fixes for proctrack/linuxproc and mpich-gm support (Takao Hatazaki, HP).
-- Fix bug in switch/elan for large task count job having irregular task
distribution across nodes.
* Changes in SLURM 0.6.9
========================
-- Fix bug in mpi plugin to set the ID correctly
-- Accounting bug causing segv fixed (Andy Riebs, 14oct.jobacct.patch)
-- Fix for failed launch of a debugged job (e.g. bad executable name).
-- Wiki plugin fix for tracking allocated nodes (Ernest Artiaga, BSC).
-- Fix memory leaks in slurmctld and federation plugin.
-- Fix sefault in federation plugin function fed_libstate_clear().
-- Align job accounting data (Andy Riebs, slurm.hp.unal_jobacct.patch)
-- Restore switch state in backup controller restarts
* Changes in SLURM 0.6.8
========================
-- Invalid AllowGroup value in slurm.conf to not cause seg fault.
-- Fix bug that would cause slurmctld to seg-fault with select/cons_res
and batch job containing more than one step.
Moe Jette
committed
* Changes in SLURM 0.6.7
========================
-- Make proctrack/linuxproc thread safe, could cause slurmd seg fault.
-- Propagate umask from srun to spawned tasks.
-- Fix problem in switch/elan error handling that could hang a slurmd
step manager process.
-- Build on AIX with -bmaxdata:0x70000000 for memory limit more than 256MB.
-- Restore srun's return code support.
Moe Jette
committed
Moe Jette
committed
* Changes in SLURM 0.6.6
========================
-- Fix for bad socket close() in the spawn-io code.
Moe Jette
committed
* Changes in SLURM 0.6.5
========================
-- Sacct to report on job steps that never actually started.
-- Added proctrack/rms to elan rpm.
-- Restructure slurmctld/agent.c logic to insure timely reaping of
terminating pthreads.
-- Srun not to hang if job fails before task launches not all completed.
Moe Jette
committed
-- Fix for consumable resources properly scheduling nodes that have more
nodes than configured (Susanne Balle, HP, cons_res_patch.10.14.2005)
Moe Jette
committed
* Changes in SLURM 0.6.4
========================
-- Bluegene plugin drains an entire bglblock on repeated boot failures
only if it has not identified a specific node as being bad.
* Changes in SLURM 0.6.3
========================
-- Fix slurmctld mem leaks (step name and hostlist struct).
-- Bluegene plugin sets end time for job terminated due to removed
bglblock.
* Changes in SLURM 0.6.2
========================
-- Fix sinfo and squeue formatting to properly handle slurm nodes,
jobs, and other names containing "%".
* Changes in SLURM 0.6.1
========================
-- Fixed smap -Db to display slurm partitions correctly (take 2).
-- Add srun fork() retry logic for very heavily loaded system.
-- Fix possible srun hang on task launch failure.
-- Add support for mvapich v0.9.4, 0.9.5 and gen2.
* Changes in SLURM 0.6.0
========================
-- Add documentation for ProctrackType=proctrack/rms.
-- Make proctrack/rms be the default for switch/elan.
-- Do not preceed SIGKILL or SIGTERM to job step with (non-requested) SIGCONT.
-- Fixed smap -Db to display slurm partitions correctly.
-- Explicitly disallow ProctrackType=proctrack/linuxproc with
SwitchType=switch/elan. They will not work properly together.
* Changes in SLURM 0.6.0-pre8
=============================
-- Remove debugging xassert in switch/federation that were accidentally
committed
-- Make slurmd step manager retry slurm_container_destroy() indefinitely
instead of giving up after 30 seconds. If something prevents a job
step's processes from being killed, the job will be stuck in the
completing until the container destroy succeeds.
* Changes in SLURM 0.6.0-pre7
=============================
-- Disable localtime_r() calls from forked processes (semaphore set
in another pthread can deadlock calls to localtime_r made from
the forked process, this will be properly fixed in the next
major release of SLURM).
-- Added SLURM_LOCALID environment variable for spawned tasks
(Dan Palermo, HP).
-- Modify switch logic to restore state based exclusively upon
recovered job steps (not state save file).
-- Gracefully refuse job if there are too many job steps in slurmd.
-- Fix race condition in job completion that can leave nodes in
COMPLETING state after job is COMPLETED.
-- Added frees for BGL BrigeAPI strdups that were to this point unknown.
-- smap scrolls correctly for BGL systems.
-- slurm_pid2jobid() API call will now return the jobid for a step
manager slurmd process.
* Changes in SLURM 0.6.0-pre6
=============================
-- Added logic to return scheduled nodes to Maui scheduler (David
Jackson, Cluster Resources)
-- Fix bug in handling job request with maximum node count.
-- Fix node selection scheduling bug with heterogeneous nodes and
srun --cpus-per-task option
-- Generate error file to note prolog failures.
* Changes in SLURM 0.6.0-pre5
=============================
-- Modify sfree (BGL command) so that --all option no longer requires
an argument.
-- Modify smap so it shows all nodes and partitions by default (even
nodes that the user can't access, otherwise there are holes in
its maps).
-- Added module to parse time string (src/common/parse_time.c) for
future use.
-- Fix BlueGene hostlist processing for non-rectangular prisms and
add string length checking.
-- Modify orphan batch job time calculation for BGL to account for
slowness when booting many bglblocks at the same time.
* Changes in SLURM 0.6.0-pre4
=============================
-- Added etc/slurm.epilog.clean to kill processes initiated outside of
slurm when a user's last job on a node terminates.
-- Added config.xml and configurator.html files for use by OSCAR.
-- Increased maximum job step count from 64 to 130 for BGL systems only.
Christopher J. Morrone
committed
=============================
-- Add code so job request for shared nodes gets explicitly requested
nodes, but lightly loaded nodes otherwise.
-- Add job step name field.
-- Add job step network specification field.
-- Add proctrack/rms plugin
-- Change the proctrack API to send a slurmd_job_t pointer to both
slurm_container_create() and slurm_container_add(). One of those
functions MUST set job->cont_id.
-- Remove vestigial node_use (virtual or coprocessor) field from job
request RPC.
-- Fix mpich-gm bugs, thanks to Takao Hatazaki (HP).
-- Fix code for clean build with gcc 2.96, Takao Hatazaki (HP).
-- Add node update state of "RESUME" to return DRAINED, DRAINING, or
DOWN node to service (IDLE or ALLOCATED state).
-- smap keeps trying to connect to slurmctld in iterative mode rather
than just aborting on failure.
-- Add squeue option --node to filter by node name.
-- Modify squeue --user option to accept not only user names, but also
user IDs.
* Changes in SLURM 0.6.0-pre2
=============================
-- Removed "make rpm" target.
Christopher J. Morrone
committed
* Changes in SLURM 0.6.0-pre1
=============================
-- Added bgl/partition_allocator/smap changes from 0.5.7.
-- Added configurable resource limit propagation (Daniel Christians, HP).