This file describes changes in recent versions of SLURM. It primarily
documents those changes that are of interest to users and admins. 

* Changes in SLURM 0.2.18
=========================
 -- Fixes for reported problems:
   - slurm/287: slurm protocol timeouts when using TotalView.
   - slurm/291: srun fails using ``-n 1'' under multi-node allocation.
   - slurm/294: srun IO buffer reports ENOSPC.
 -- Memory corruption bug fixed, it was causing slurmctld to seg-fault
 -- Non-responding nodes now go from DRAINING to DRAINED state when 
    jobs complete
 -- Do not schedule pending jobs while any job is actively COMPLETING 
    unless the submitted job specifically identifies its nodes (like DPCS)
 -- Reset priority of jobs with priority==1 when a non-responding node 
    starts to respond again
 -- Ignore jobs with priority==1 when establishing new baseline upon 
    slurmctld restart
 -- Make slurmctld/message retry be timer based rather than queue based 
    for better scalability
 -- Slurmctld logging is more concise, using hostlists more
 -- srun --no-allocate used special job_id range to avoid conflicts 
    or premature job termination (purging by slurmctld)
 -- New --jobid=id option in srun to initiate job step under an existing 
    allocation.
 -- Support in srun for TotalView bulk launch.

* Changes in SLURM 0.2.17
=========================
 -- Fixes for reported problems:
   - slurm/279: Hold jobs that can't execute due to DOWN or DRAINED 
     nodes and release when nodes are returned to service.
   - slurm/285: "srun killed due to SIGPIPE"
 -- Support for running job steps on nodes relative to current 
    allocation via srun -r, --relative=n option.
 -- SIGKILL no longer broadcasted to job via srun on task failure unless
    --no-allocate option is used.
 -- Re-enabled "chkconfig --add" in default RPMs.
 -- Backup controller setting proper PID into slurmctld.pid file.
 -- Backup controller restores QSW state each time it assumes control
 -- Backup controller purges old job records before assuming control
    to avoid resurrecting defunct jobs.
 -- Kill jobs on non-responding DRAINING nodes and make their state
    DRAINED.
 -- Save state upon completion of a job's last EPILOG_COMPLETION to 
    reduce possibility of inconsistent job and node records when the 
    controller is transitioning between primary and backup. 
 -- Change logging level of detailed communication errors to not print 
    them unless detailed debugging is requested.
 -- Increase number of concurrent controller server threads from 20 
    to 50 and restructure code to handle backlogs more efficiently.
 -- Partition state at controller startup is based upon slurm.conf 
    rather than previously saved state. Additional improvements to 
    avoid inconsistent job/node/partition states at restart. Job state 
    information is used to arbitrate conflicts.
 -- Orphaned file descriptors eliminated.

* Changes in SLURM 0.2.16
=========================
 -- Fixes for reported problems:
   - slurm/265: Early termination of srun could cause job to remain in queue.
   - slurm/268: Slurmctld could deadlock if there was a delay in the 
     termination of a large node-count job. An EPILOG_COMPLETE RPC was 
     added so that slurmd could notify slurmctld whenever the job 
     termination was completed.
   - slurm/270: Segfault in sinfo if a configured node lacked a partition.
   - slurm/278: Exit code in scontrol did not indicate failure.
 -- Fixed bug in slurmd that caused the daemon to occaisionally kill itself.
 -- Fixed bug in srun when running with --no-allocate and >1 process per node.
 -- Small fixes and updates for srun manual.

* Changes in SLURM 0.2.15
=========================
 -- Fixes for reported problems:
   - slurm/265: Job was orphaned when allocation response message could 
     not be sent. Job is now killed on allocation response message transmit 
     failure and socket error details are logged.
   - Fix for slurm/267: "Job epilog may run multiple times."
 -- Squeue job TIMELIMIT format changed from "h:mm" to "d:h:mm:ss".
 -- DPCS initiated jobs have steps execute properly without explicit 
    specification of node count.

* Changes in SLURM 0.2.14
=========================
 -- Fixes for reported problems:
   - slurm/194: "srun doesn't handle most options when run under an allocation."
   - slurm/244: "REQ: squeue shows requested size of pending jobs."
 -- SLURM_NODELIST environment variable now exported to all jobs, not
    only batch jobs.
 -- Nodelist displayed in squeue for completing jobs is now restricted to 
    completing nodes.
 -- Node "reason" field properly displayed in sinfo even with filtering. 
 -- ``slurm_tv_clean'' daemon now supports a log file.
 -- Batch jobs are now re-queued on launch failure.
 -- Controller confirms job scripts for batch jobs are still running on 
    node zero at node registration.
 -- Default RPMs no longer stop/start SLURM daemons on upgrade or install.

* Changes in SLURM 0.2.13
=========================
 -- Fixes for reported problems:
   - Fixed bug in slurmctld where "drained" nodes would go back into
     the "idle" state under some conditions (slurm/228).
   - Added possible fix for slurm/229: "slurmd occasionally fails
     to reap all children."
 -- Fixed memory leak in auth_munge plugin.
 -- Added fix to slurmctld to allow arbitrarily large job specifications
    to be saved and recovered in the state file.
 -- Allow "updates" in the configuration file of previously defined
    node state and reason. 
 -- On "forceful termination" of a running job step, srun now exits
    unconditionally, instead of waiting for all I/O.
 -- Slurmctld now uses pidfile to kill old daemon when a new one is started.
 -- Addition of new daemon "slurm_tv_clean" used to clean up jobs orphaned
    due to use of the TotalView parallel debugger.

* Changes in SLURM 0.2.12
=========================
 -- Fixes for reported problems:
   - Fix for "waitpid: No child processes" when using TotalView (slurm/217).
   - Implemented temporary workaround for slurm/223: "Munge decode failed: 
     Munged communication error." 
   - Temporary fix for slurm/222: "elan3_create(0): Invalid argument."
 -- Fixed memory leaks in slurmctld (mostly due to reconfigure).
 -- More squeue/sinfo interface changes (see squeue(1), sinfo(1)).
 -- Sinfo now accepts list of node states to -t,--state option.
 -- Node "reason" field now available via sinfo command (see sinfo(1)).
 -- Wrapper source for srun (srun.wrapper.c) now installed and available
    for TotalView support.
 -- Improved retry login in user commands for periods when slurmctld
    primary is down and backup has not yet taken over.

* Changes in SLURM 0.2.11
=========================
 -- Changes in srun:
   - Fixed bug in signal handling that occaisonally resulted in orphaned 
     jobs when using Ctrl-C.
   - Return non-zero exit code when remote tasks are killed by a signal.
   - SIGALRM is now blocked by default.
 -- Added ``reason'' string for down, drained, or draining nodes. 
 -- Added -V,--version option to squeue and sinfo.
 -- Improved some error messages from user utilities.

* Changes in SLURM 0.2.10
=========================
 -- New slurm.conf configuration parameters:
   - WaitTime:    Default for srun -w,--wait parameter.
   - MaxJobCount: Maximum number of jobs SLURM can handle at one time.
   - MinJobAge:   Minimum time since completing before job is purged from 
                  slurmctld memory.
 -- Block user defined signals USR1 and USR2 in slurmd session manager.
 -- More squeue cleanup.
 -- Support for passing options to sinfo via environment variables.
 -- Added option to scontrol to find intersection of completing jobs and nodes.
 -- Added fix in auth_munge to prevent "Munged communication error" message.

* Changes in SLURM 0.2.9
========================
 -- Fixes for reported problems:
   - Argument to srun `-n' option was taken as octal if preceeded with a `0'.
 -- New format for Elan hosts config file (/etc/elanhosts. See README)
 -- Various fixes for managing COMPLETING jobs.
 -- Support for passing options to squeue via environment variables 
    (see squeue(1))

* Changes in SLURM 0.2.8
=========================
 -- Fix for bug in slurmd that could make debug messages appear in job output.
 -- Fix for bug in slurmctld retry count computation.
 -- Srun now times out slow launch threads.
 -- "Time Used" output in squeue now includes seconds.

* Changes in SLURM 0.2.7
=========================
 -- Fix for bug in Elan module that results in slurmd hang.
 -- Added completing job state to default list of states to print with squeue.

* Changes in SLURM 0.2.6
=========================
 -- More fixes for handling cleanup of slow terminating jobs.
 -- Fixed bug in srun that might leave nodes allocated after a Ctrl-C.

* Changes in SLURM 0.2.5
=========================
 -- Various fixes for cleanup of slow terminating or unkillable jobs.
 -- Fixed some small memory leaks in communications code.
 -- Added hack for synchronized exit of jobs on large node count.
 -- Long lists of nodes are no longer truncated in sinfo.
 -- Print more descriptive error message when tasks exit with nonzero status.
 -- Fixed bug in srun where unsuccessful launch attempts weren't detected.
 -- Elan network error resolver thread now runs from elan module in slurmd.
 -- Slurmctld uses consecutive Elan context and program description numbers
    instead of choosing them randomly.

* Changes in SLURM 0.2.4
==========================
 -- Fix for file descriptor leak in slurmctld.
 -- auth_munge plugin now prints credential info on decode failure.
 -- Minor changes to scancel interface.
 -- Filename format option "%J" now works again for srun --output and --error.
 
* Changes in SLURM 0.2.3
==========================
 -- Fix bug in srun when using per-task files for stderr.
 -- Better error reporting on failure to open per-task input/output files.
 -- Update auth_munge plugin for munge 0.1.
 -- Minor changes to squeue interface.
 -- New srun option `--hold' to submit job in "held" state.

* Changes in SLURM 0.2.2
==========================
 -- Fixes for reported problems:
   - Execution of script allocate mode fails in some cases. (gnats:161)
   - Errors using per-task input files with Elan support. (gnats:162)
   - srun doesn't handle all environment variables properly. (gnats:164)
 -- Parallel job is now terminated if a task is killed by a signal.
 -- Exit status of srun is set based on exit codes of tasks.
 -- Redesign of sinfo interface and options.
 -- Shutdown of slurmctld no longer propagates shutdown to all nodes.

* Changes in SLURM 0.2.1
===========================
 -- Fix bug where reconfigure request to slurmctld killed the daemon.

* Changes in SLURM 0.2.0
============================

 -- SlurmdTimeout of 0 means never set a non-responding node to DOWN.
 -- New srun option, -u,--unbuffered, for unbuffered stdout.
 -- Enhancements for sinfo
   - Non-responding nodes show "*" character appended instead of "NoResp+".
   - Node states show abbreviated variant by default
 -- Enhancements for scontrol.
   - Added "ping" command to show current state of SLURM controllers.
   - Job dump in scontrol shows user name as well as UID. 
   - Node state of DRAIN is appropriately mapped to DRAINING or DRAINED.
 -- Fix for bug where request for task count greater than partition limit
    was queued anyway.
 -- Fix for bugs in job end time handling.
 -- Modifications for error free builds on 64 bit architectures.
 -- Job cancel immediately deallocates nodes instead of waiting on srun.
 -- Attempt to create slurmd spool if it does not exist.
 -- Fixed signal handling bug in srun allocate mode.
 -- Earlier error detection in slurmd startup.
 -- "fatal: _shm_unlock: Numerical result out of range" bug fixed in slurmd.
 -- Config file parsing is now case insensitive.
 -- SLURM_NODELIST environment variable now set in allocate mode.

* Changes in SLURM 0.2.0-pre2
=============================

 -- Fix for reconfigure when public/private key path is changed.
 -- Shared memory fixes in slurmd. 
   - fix for infinite semaphore incrementation bug.
 -- Semaphore fixes in slurmctld.
 -- Slurmctld now remembers which nodes have registered after recover.
 -- Fixed reattach bug when tasks have exited.
 -- Change directory to /tmp in slurmd if daemonizing.
 -- Logfiles are reopened on reconfigure.

$Id$