NEWS 271 KB
Newer Older
Danny Auble's avatar
Danny Auble committed
* Changes in Slurm 14.03.0pre2
==============================
 -- Added "JobAcctGatherParams" configuration parameter. Value of "NoShare"
    disables accounting for shared memory.
 -- Added fields to "scontrol show job" output: boards_per_node,
    sockets_per_board, ntasks_per_node, ntasks_per_board, ntasks_per_socket,
    ntasks_per_core, and nice.
 -- Add squeue output format options for job command and working directory
    (%o and %Z respectively).
 -- Add stdin/out/err to sview job output.
 -- Add new job_state of JOB_BOOT_FAIL for job terminations due to failure to
    boot it's allocated nodes or BlueGene block.
 -- CRAY - Add SelectTypeParameters NHC_NO_STEPS and NHC_NO which will disable
    the node health check script for steps and allocations respectfully.
 -- Reservation with CoreCnt: Avoid possible invalid memory reference.
 -- Add new error code for attempt to create a reservation with duplicate name.
 -- Validate that a hostlist file contains text (i.e. not a binary).
 -- switch/generic - propagate switch information from srun down to slurmd and
    slurmstepd.
 -- CRAY - Do not package Slurm's libpmi or libpmi2 libraries. The Cray version
    of those libraries must be used.
 -- Added a new option to the scontrol command to view licenses that are
    configured in use and avalable. 'scontrol show licenses'.
 -- MySQL - Made Slurm compatible with 5.6
Danny Auble's avatar
Danny Auble committed
* Changes in Slurm 14.03.0pre1
 -- sview - improve scalability
 -- Add task pointer to the task_post_term() function in task plugins. The
    terminating task's PID is available in task->pid.
 -- Move select/cray to select/alps
 -- Defer sending SIGKILL signal to processes while core dump in progress.
 -- Added JobContainerPlugin configuration parameter and plugin infrastructure.
 -- Added partition configuration parameters AllowAccounts, AllowQOS,
 -- The rpmbuild option for a cray system with ALPS has changed from
    %_with_cray to %_with_cray_alps.
 -- The log file timestamp format can now be selected at runtime via the
    LogTimeFormat configuration option. See the slurm.conf and slurmdbd.conf
 -- Added switch/generic plugin to a job's convey network topology.
 -- BLUEGENE - If block is in 'D' state or has more cnodes in error than
    MaxBlockInError set the job wait reason appropriately.
 -- API use: Generate an error return rather than fatal error and exit if the
    configuraiton file is absent or invalid. This will permit Slurm APIs to be
    more reliably used by other programs.
 -- Add support for load-based scheduling, allocate jobs to nodes with the
    largest number of available CPUs. Added SchedulingParameters paramter of
    "CR_LLN" and partition parameter of "LLN=yes|no".
 -- Added job_info() and step_info() functions to the gres plugins to extract
    plugin specific fields from the job's or step's GRES data structure.
 -- Added sbatch --signal option of "B:" to signal the batch shell rather than
    only the spawned job steps.
 -- Added sinfo and squeue format option of "%all" to print all fields available
    for the data type with a vertical bar separating each field.
 -- Add mechanism for job_submit plugin to generate error message for srun,
Morris Jette's avatar
Morris Jette committed
    salloc or sbatch to stderr. New argument added to job_submit function in
 -- Add StdIn, StdOut, and StdErr paths to job information dumped with
    "scontrol show job".
 -- Permit Slurm administrator to submit a batch job as any user.
 -- Set a job's RLIMIT_AS limit based upon it's memory limit and VsizeFactor
    configuration value.
 -- Make jobacct_gather/cgroup work correctly and also make all jobacct_gather
    plugins more maintainable.
 -- Proctrack/pgid - Add support for proctrack_p_plugin_get_pids() function.
 -- Sched/backfill - Change default max_job_bf parameter from 50 to 100.
 -- Added -I|--item-extract option to sh5util to extract data item from series.
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.6.10
=========================
 -- Switch/nrt - On switch resource allocation failure, free partial allocation.
 -- Switch/nrt - Properly track usage of CAU and RDMA resources with multiple
    tasks per compute node.
 -- Fix issue where user is requesting --acctg-freq=0 and no memory limits.
 -- BGQ - Temp fix issue where job could be left on job_list after it finished.
 -- BGQ - Fix issue where limits were checked on midplane counts instead of
    cnode counts.
 -- BGQ - Move code to only start job on a block after limits are checked.
 -- Handle node ranges better when dealing with accounting max node limits.
 -- Fix perlapi to compile correctly with perl 5.18
 -- BGQ - Fix issue with uninitialized variable.
 -- Correct sinfo --sort fields to match documentation: E => Reason,
    H -> Reason Time (new), R -> Partition Name, u/U -> Reason user (new)
 -- If an invalid assoc_ptr comes in don't use the id to verify it.
 -- Sched/backfill modified to avoid using nodes in completing state.
 -- Correct support for job --profile=none option and related documentation.
 -- Properly enforce job --requeue and --norequeue options.
 -- If a job --mem-per-cpu limit exceeds the partition or system limit, then
    scale the job's memory limit and CPUs per task to satisfy the limit.
Morris Jette's avatar
Morris Jette committed
 -- Correct logic to support Power7 processor with 1 or 2 threads per core
    (CPU IDs are not consecutive).
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.6.9
========================
 -- Fix sinfo to work correctly with draining/mixed nodes as well as filtering
    on Mixed state.
 -- Fix sacctmgr update user with no "where" condition.
 -- Fix logic bugs for SchedulerParameters option of max_rpc_cnt.
* Changes in Slurm 2.6.8
========================
 -- Add support for Torque/PBS job array options and environment variables.
 -- CRAY/ALPS - Add support for CLE52
 -- Fix issue where jobs still pending after a reservation would remain
    in waiting reason ReqNodeNotAvail.
 -- Update last_job_update when a job's state_reason was modified.
 -- Free job_ptr->state_desc where ever state_reason is set.
 -- Fixed sacct.1 and srun.1 manual pages which contains a hyphen where
    a minus sign for options was intended.
 -- sinfo - Make sure if partition name is long and the default the last char
    doesn't get chopped off.
 -- task/affinity - Protect against zero divide when simulating more hardware
    than you really have.
 -- NRT - Fix issue with 1 node jobs.  It turns out the network does need to
    be setup for 1 node jobs.
 -- Fix recovery of job dependency on task of job array when slurmctld restarts.
 -- mysql - Fix invalid memory reference.
 -- Lock the /cgroup/freezer subsystem when creating files for tracking processes.
 -- Fix preempt/partition_prio to avoid preempting jobs in partitions with
    PreemptMode=OFF
 -- launch/poe - Implicitly set --network in job step create request as needed.
 -- Permit multiple batch job submissions to be made for each run of the
    scheduler logic if the job submissions occur at the nearly same time.
 -- Fix issue where associations weren't correct if backup takes control and
    new associations were added since it was started.
Danny Auble's avatar
Danny Auble committed
 -- Fix race condition is corner case with backup slurmctld.
 -- With the backup slurmctld make sure we reinit beginning values in the
    slurmdbd plugin.
 -- Fix sinfo to work correctly with draining/mixed nodes.
 -- MySQL - Fix it so a lock isn't held unnecessarily.
 -- Added new SchedulerParameters option of max_rpc_cnt when too many RPCs
    are active.
 -- BGQ - Fix deny_pass to work correctly.
 -- BGQ - Fix sub block steps using a block when the block has passthrough's
    in it.
* Changes in Slurm 2.6.7
========================
 -- Properly enforce a job's cpus-per-task option when a job's allocation is
    constrained on some nodes by the mem-per-cpu option.
 -- Correct the slurm.conf man pages and checkpoint_blcr.html page
    describing that jobs must be drained from cluster before deploying
Morris Jette's avatar
Morris Jette committed
    any checkpoint plugin. Corrected in version 14.03.
 -- Fix issue where if using munge and munge wasn't running and a slurmd
Morris Jette's avatar
Morris Jette committed
    needed to forward a message, the slurmd would core dump.
 -- Update srun.1 man page documenting the PMI2 support.
 -- Fix slurmctld core dump when a jobs gets its QOS updated but there
    is not a corresponding association.
 -- If a job requires specific nodes and can not run due to those nodes being
    busy, the main scheduling loop will block those specific nodes rather than
    the entire queue/partition.
 -- Fix minor memory leak when updating a job's name.
 -- Fix minor memory leak when updating a reservation on a partition using "ALL"
    nodes.
 -- Fix minor memory leak when adding a reservation with a nodelist and core
    count.
 -- Update sacct man page description of job states.
 -- BGQ - Fix minor memory leak when selecting blocks that can't immediately be
    placed.
 -- Fixed minor memory leak in backfill scheduler.
 -- MYSQL - Fixed memory leak when querying clusters.
 -- MYSQL - Fix when updating QOS on an association.
 -- NRT - Fix to supply correct error messages to poe/pmd when a launch fails.
 -- Add SLURM_STEP_ID to Prolog environment.
 -- Add support for SchedulerParameters value of bf_max_job_start that limits
    the total number of jobs that can be started in a single iteration of the
    backfill scheduler.
 -- Don't print negative number when dealing with large memory sizes with
    sacct.
 -- Fix sinfo output so that host in state allocated and mixed will not be
    merged together.
 -- GRES: Avoid crash if GRES configurations is inconstent.
 -- Make S_SLURM_RESTART_COUNT item available to SPANK.
Morris Jette's avatar
Morris Jette committed
 -- Munge plugins - Add sleep between retries if can't connect to socket.
 -- Fix the database query to return all pending jobs in a given time interval.
 -- switch/nrt - Correct logic to get dynamic window count.
 -- Remove need to use job->ctx_params in the launch plugin, just to simplify
    code.
 -- NRT - Fix possible memory leak if using multiple adapters.
 -- NRT - Fix issue where there are more than NRT_MAXADAPTERS on a system.
 -- NRT - Increase Max number of adapters from 8 -> 9
 -- NRT - Initialize missing variables when the PMD is starting a job.
 -- NRT - Fix issue where we are launching hosts out of numerical order,
    this would cause pmd's to hang.
 -- NRT - Change xmalloc's to malloc just to be safe.
 -- NRT - Sanity check to make sure a jobinfo is there before packing.
 -- Add missing options to the print of TaskPluginParam.
 -- Fix a couple of issues with scontrol reconfig and adding nodes to
    slurm.conf.  Rebooting daemons after adding nodes to the slurm.conf
    is highly recommended.
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.6.6
========================
 -- sched/backfill - Fix bug that could result in failing to reserve resources
    for high priority jobs.
 -- Correct job RunTime if requeued from suspended state.
 -- Reset job priority from zero (held) on manual resume from suspend state.
 -- If FastSchedule=0 then do not DOWN a node with low memory or disk size.
Danny Auble's avatar
Danny Auble committed
 -- Remove vestigial note.
 -- Update sshare.1 man page making it consistent with sacctmgr.1.
 -- Do not reset a job's priority when the slurmctld restarts if previously
    set to some specific value.
 -- sview - Fix regression where the Node tab wasn't able to add/remove columns.
 -- Fix slurmstepd lock when job terminates inside the infiniband
    network traffic accounting plugin.
 -- Correct the documentation to read filesystem instead of Lustre. Update
    the srun help.
 -- Fix the acct_gather_filesystem_lustre.c to compute the Lustre accounting
    data correctly accumulating differences between sampling intervals.
    Fix the data structure mismatch between acct_gather_filesystem_lustre.c
    and slurm_jobacct_gather.h which caused the hdf5 plugin to log incorrect
    data.
 -- Don't allow PMI_TIME to be zero which will cause floating exception.
 -- Fix purging of old reservation errors in database.
 -- MYSQL - If starting the plugin and the database isn't up attempt to
    connect in a loop instead of producing a fatal.
 -- BLUEGENE - If IONodesPerMP changes in bluegene.conf recalculate bitmaps
    based on ionode count correctly on slurmctld restart.
 -- Fix step allocation when some CPUs are not available due to memory limits.
    This happens when one step is active and using memory that blocks the
    scheduling of another step on a portion of the CPUs needed. The new step
    is now delayed rather than aborting with "Requested node configuration is
    not available".
 -- Make sure node limits get assessed if no node count was given in request.
 -- Removed obsolete slurm_terminate_job() API.
 -- Update documentation about QOS limits
 -- Retry task exit message from slurmstepd to srun on message timeout.
 -- Correction to logic reserving all nodes in a specified partition.
Danny Auble's avatar
Danny Auble committed
 -- Added support for selecting AMD GPU by setting GPU_DEVICE_ORDINAL env var.
 -- Properly enforce GrpSubmit limit for job arrays.
 -- CRAY - fix issue with using CR_ONE_TASK_PER_CORE
 -- CRAY - fix memory leak when using accelerators
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.6.5
========================
 -- Correction to hostlist parsing bug introduced in v2.6.4 for hostlists with
    more than one numeric range in brackets (e.g. rack[0-3]_blade[0-63]").
 -- Add notification if using proctrack/cgroup and task/cgroup when oom hits.
 -- Corrections to advanced reservation logic with overlapping jobs.
 -- job_submit/lua - add cpus_per_task field to those available.
 -- Add cpu_load to the node information available using the Perl API.
 -- Correct a job's GRES allocation data in accounting records for non-Cray
    systems.
 -- Substantial performance improvement for systems with Shared=YES or FORCE
    and large numbers of running jobs (replace bubble sort with quick sort).
 -- proctrack/cgroup - Add locking to prevent race condition where one job step
    is ending for a user or job at the same time another job stepsis starting
    and the user or job container is deleted from under the starting job step.
 -- Fixed sh5util loop when there are no node-step files.
 -- Fix race condition on batch job termination that could result in a job exit
    code of 0xfffffffe if the slurmd on node zero registers its active jobs at
    the same time that slurmstepd is recording the job's exit code.
 -- Correct logic returning remaining job dependencies in job information
    reported by scontrol and squeue. Eliminates vestigial descriptors with
    no job ID values (e.g. "afterany").
 -- Improve performance of REQUEST_JOB_INFO_SINGLE RPC by removing unnecessary
    locks and use hash function to find the desired job.
 -- jobcomp/filetxt - Reopen the file when slurmctld daemon is reconfigured
    or gets SIGHUP.
 -- Remove notice of CVE with very old/deprecated versions of Slurm in
    news.html.
 -- Fix if hwloc_get_nbobjs_by_type() returns zero core count (set to 1).
 -- Added ApbasilTimeout parameter to the cray.conf configuration file.
 -- Handle in the API if parts of the node structure are NULL.
 -- Fix srun hang when IO fails to start at launch.
jette's avatar
jette committed
 -- Fix for GRES bitmap not matching the GRES count resulting in abort
    (requires manual resetting of GRES count, changes to gres.conf file,
    and slurmd restarts).
 -- Modify sview to better support job arrays.
 -- Modify squeue to support longer job ID values (for many job array tasks).
 -- Fix race condition in authentication credential creation that could corrupt
    memory. (NOTE: This race condition has existed since 2003 and would be
    exceedingly rare.)
 -- HDF5 - Fix minor memory leak.
 -- Slurmstepd variable initialization - Without this patch, free() is called
    on a random memory location (i.e. whatever is on the stack), which can
    result in slurmstepd dying and a completed job not being purged in a
    timely fashion.
 -- Fix slurmstepd race condition when separate threads are reading and
    modifying the job's environment, which can result in the slurmstepd failing
    with an invalid memory reference.
 -- Fix erroneous error messages when running gang scheduling.
 -- Fix minor memory leak.
 -- scontrol modified to suspend, resume, hold, uhold, or release multiple
    jobs in a space separated list.
 -- Minor debug error when a connection goes away at the end of a job.
 -- Validate return code from calls to slurm_get_peer_addr
 -- BGQ - Fix issues with making sure all cnodes are accounted for when mulitple
    steps cause multiple cnodes in one allocation to go into error at the
    same time.
 -- scontrol show job - Correct NumNodes value calculated based upon job
    specifications.
 -- BGQ - Fix issue if user runs multiple sub-block jobs inside a multiple
    midplane block that starts on a higher coordinate than it ends (i.e if a
    block has midplanes [0010,0013] 0013 is the start even though it is
    listed second in the hostlist).
 -- BGQ - Add midplane to the total_cnodes used in the runjob_mux plugin
    for better debug.
 -- Update AllocNodes paragraph in slurm.conf.5.
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.6.4
========================
 -- Fixed sh5util to print its usage.
 -- Corrected commit f9a3c7e4e8ec.
 -- Honor ntasks-per-node option with exclusive node allocations.
 -- sched/backfill - Prevent invalid memory reference if bf_continue option is
    configured and slurm is reconfigured during one of the sleep cycles or if
    there are any changes to the partition configuration or if the normal
    scheduler runs and starts a job that the backfill scheduler is actively
    working on.
 -- Update man pages information about acct-freq and JobAcctGatherFrequency
    to reflect only the latest supported format.
 -- Minor document update to include note about PrivateData=Usage for the
    slurm.conf when using the DBD.
 -- Expand information reported with DebugFlags=backfill.
 -- Initiate jobs pending to run in a reservation as soon as the reservation
    becomes active.
 -- Purged expired reservation even if it has pending jobs.
 -- Corrections to calculation of a pending job's expected start time.
 -- Remove some vestigial logic treating job priority of 1 as a special case.
 -- Memory freeing up to avoid minor memory leaks at close of daemons
 -- Updated documentation to give correct units being displayed.
 -- Report AccountingStorageBackupHost with "scontrol show config".
 -- init scripts ignore quotes around Pid file name specifications.
 -- Fixed typo about command case in quickstart.html.
 -- task/cgroup - handle new cpuset files, similar to commit c4223940.
 -- Replace the tempname() function call with mkstemp().
Morris Jette's avatar
Morris Jette committed
 -- Fix for --cpu_bind=map_cpu/mask_cpu/map_ldom/mask_ldom plus
    --mem_bind=map_mem/mask_mem options, broken in 2.6.2.
 -- Restore default behavior of allocating cores to jobs on a cyclic basis
    across the sockets unless SelectTypeParameters=CR_CORE_DEFAULT_DIST_BLOCK
    or user specifies other distribution options.
 -- Enforce JobRequeue configuration parameter on node failure. Previously
    always requeued the job.
 -- acct_gather_energy/ipmi - Add delay before retry on read error.
 -- select/cons_res with GRES and multiple threads per core, fix possible
    infinite loop.
 -- proctrack/cgroup - Add cgroup create retry logic in case one step is
    starting at the same time as another step is ending and the logic to create
    and delete cgroups overlaps.
 -- Improve setting of job wait "Reason" field.
 -- Correct sbatch documentation and job_submit/pbs plugin "%j" is job ID,
    not "%J" (which is job_id.step_id).
 -- Improvements to sinfo performance, especially for large numbers of
    partitions.
 -- SlurmdDebug - Permit changes to slurmd debug level with "scontrol reconfig"
 -- smap - Avoid invalid memory reference with hidden nodes.
 -- Fix sacctmgr modify qos set preempt+/-=.
 -- BLUEGENE - fix issue where node count wasn't set up correctly when srun
    preforms the allocation, regression in 2.6.3.
 -- Add support for dependencies of job array elements (e.g.
    "sbatch --depend=afterok:123_4 ...") or all elements of a job array (e.g.
    "sbatch --depend=afterok:123 ...").
Morris Jette's avatar
Morris Jette committed
 -- Add support for new options in sbatch qsub wrapper:
    -W block=true	(wait for job completion)
    Clear PBS_NODEFILE environment variable
David Bigagli's avatar
David Bigagli committed
 -- Fixed the MaxSubmitJobsPerUser limit in QOS which limited submissions
    a job too early.
 -- sched/wiki, sched/wiki2 - Fix to work with change logic introduced in
    version 2.6.3 preventing Maui/Moab from starting jobs.
 -- Updated the QOS limits documentation and man page.
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.6.3
========================
 -- Add support for some new #PBS options in sbatch scripts and qsub wrapper:
    -l accelerator=true|false	(GPU use)
    -l mpiprocs=#	(processors per node)
    -l naccelerators=#	(GPU count)
    -l select=#		(node count)
    -l ncpus=#		(task count)
    -v key=value	(environment variable)
    -W depend=opts	(job dependencies, including "on" and "before" options)
    -W umask=#		(set job's umask)
Eric Winter's avatar
Eric Winter committed
 -- Added qalter and qrerun commands to torque package.
 -- Corrections to qstat logic: job CPU count and partition time format.
 -- Add job_submit/pbs plugin to translate PBS job dependency options to the
    extend possible (no support for PBS "before" options) and set some PBS
    environment variables.
 -- Add spank/pbs plugin to set a bunch of PBS environment variables.
 -- Backported sh5util from master to 2.6 as there are some important
    bugfixes and the new item extraction feature.
Morris Jette's avatar
Morris Jette committed
 -- select/cons_res - Correct MacCPUsPerNode partition constraint for CR_Socket.
Morris Jette's avatar
Morris Jette committed
 -- scontrol - for setdebugflags command, avoid parsing "-flagname" as an
    scontrol command line option.
 -- Fix issue with step accounting if a job is requeued.
 -- Close file descriptors on exec of prolog, epilog, etc.
 -- Fix issue when a user has held a job and then sets the begin time
    into the future.
 -- Scontrol - Enable changing a job's stdout file.
 -- Fix issues where memory or node count of a srun job is altered while the
    srun is pending.  The step creation would use the old values and possibly
    hang srun since the step wouldn't be able to be created in the modified
    allocation.
 -- Add support for new SchedulerParameters value of "bf_max_job_part", the
    maximum depth the backfill scheduler should go in any single partition.
 -- acct_gather/infiniband plugin - Correct packets_in/out values.
 -- BLUEGENE - Don't ignore a conn-type request from the user.
 -- BGQ - Force a request on a Q for a MESH to be a TORUS in a dimension that
    can only be a TORUS (1).
 -- Change max message length from 100MB to 1GB before generating "Insane
    message length" error.
 -- sched/backfill - Prevent possible memory corruption due to use of
    bf_continue option and long running scheduling cycle (pending jobs could
    have been cancelled and purged).
 -- CRAY - fix AcceleratorAllocation depth correctly for basil 1.3
 -- Created the environment variable SLURM_JOB_NUM_NODES for srun jobs and
    updated the srun man page.
 -- BLUEGENE/CRAY - Don't set env variables that pertain to a node when Slurm
    isn't doing the launching.
 -- gres/gpu and gres/mic - Do not treat the existence of an empty gres.conf
    file as a fatal error.
 -- Fixed for if hours are specified as 0 the time days-0:min specification
    is not parsed correctly.
 -- switch/nrt - Fix for memory leak.
Morris Jette's avatar
Morris Jette committed
 -- Subtract the PMII_COMMANDLEN_SIZE in contribs/pmi2/pmi2_api.c to prevent
    certain implementation of snprintf() to segfault.
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.6.2
========================
 -- Fix issue with reconfig and GrpCPURunMins
 -- Fix of wrong node/job state problem after reconfig
Danny Auble's avatar
Danny Auble committed
 -- Allow users who are coordinators update their own limits in the accounts
    they are coordinators over.
 -- BackupController - Make sure we have a connection to the DBD first thing
    to avoid it thinking we don't have a cluster name.
 -- Correct value of min_nodes returned by loading job information to consider
    the job's task count and maximum CPUs per node.
 -- If running jobacct_gather/none fix issue on unpacking step completion.
 -- Reservation with CoreCnt: Avoid possible invalid memory reference.
 -- sjstat - Add man page when generating rpms.
 -- Make sure GrpCPURunMins is added when creating a user, account or QOS with
    sacctmgr.
 -- Fix for invalid memory reference due to multiple free calls caused by
    job arrays submitted to multiple partitions.
 -- Enforce --ntasks-per-socket=1 job option when allocating by socket.
 -- Validate permissions of key directories at slurmctld startup. Report
    anything that is world writable.
 -- Improve GRES support for CPU topology. Previous logic would pick CPUs then
    reject jobs that can not match GRES to the allocated CPUs. New logic first
    filters out CPUs that can not use the GRES, next picks CPUs for the job,
    and finally picks the GRES that best match those CPUs.
 -- Switch/nrt - Prevent invalid memory reference when allocating single adapter
    per node of specific adapter type
 -- CRAY - Make Slurm work with CLE 5.1.1
 -- Fix segfault if submitting to multiple partitions and holding the job.
 -- Use MAXPATHLEN instead of the hardcoded value 1024 for maximum file path
    lengths.
 -- If OverTimeLimit is defined do not declare failed those jobs that ended
    in the OverTimeLimit interval.
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.6.1
========================
 -- slurmdbd - Allow job derived ec and comments to be modified by non-root
    users.
 -- Fix issue with job name being truncated to 24 chars when sending a mail
    message.
 -- Fix minor issues with spec file, missing files and including files
    erroneously on a bluegene system.
 -- sacct - fix --name and --partition options when using
    accounting_storage/filetxt.
 -- squeue - Remove extra whitespace of default printout.
 -- BGQ - added head ppcfloor as an include dir when building.
 -- BGQ - Better debug messages in runjob_mux plugin.
David Bigagli's avatar
David Bigagli committed
 -- PMI2 Updated the Makefile.am to build a versioned library.
 -- CRAY - Fix srun --mem_bind=local option with launch/aprun.
 -- PMI2 Corrected buffer size computation in the pmi2_api.c module.
 -- GRES accounting data wrong in database: gres_alloc, gres_req, and gres_used
    fields were empty if the job was not started immediately.
 -- Fix sbatch and srun task count logic when --ntasks-per-node specified,
    but no explicit task count.
 -- Corrected the hdf5 profile user guide and the acct_gather.conf
    documentation.
 -- IPMI - Fix Math bug getting new wattage.
David Bigagli's avatar
David Bigagli committed
 -- Corrected the AcctGatherProfileType documentation in slurm.conf
 -- Corrected the sh5util program to print the header in the csv file
    only once, set the debug messages at debug() level, make the argument
David Bigagli's avatar
David Bigagli committed
    check case insensitive and avoid printing duplicate \n.
 -- If cannot collect energy values send message to the controller
    to drain the node and log error slurmd log file.
 -- Handle complete removal of CPURunMins time at the end of the job instead
    of at multifactor poll.
 -- sview - Add missing debug_flag options.
 -- PGSQL - Notes about Postgres functionality being removed in the next
    version of Slurm.
 -- MYSQL - fix issue when rolling up usage and events happened when a cluster
    was down (slurmctld not running) during that time period.
 -- sched/wiki2 - Insure that Moab gets current CPU load information.
 -- Prevent infinite loop in parsing configuration if including file containing
    one blank line.
 -- Fix pack and unpack between 2.6 and 2.5.
 -- Fix job state recovery logic in which a job's accounting frequency was
    not set. This would result in a value of 65534 seconds being used (the
    equivalent of NO_VAL in uint16_t), which could result in the job being
    requeued or aborted.
 -- Validate a job's accounting frequency at submission time rather than
    waiting for it's initiation to possibly fail.
 -- Fix CPURunMins if a job is requeued from a failed launch.
 -- Fix in accounting_storage/filetxt to correct start times which sometimes
    could end up before the job started.
 -- Fix issue with potentially referencing past an array in parse_time()
 -- CRAY - fix issue with accelerators on a cray when parsing BASIL 1.3 XML.
 -- Fix issue with a 2.5 slurmstepd locking up when talking to a 2.6 slurmd.
 -- Add argument to priority plugin's priority_p_reconfig function to note
    when the association and QOS used_cpu_run_secs field has been reset.
* Changes in Slurm 2.6.0
========================
 -- Fix it so bluegene and serial systems don't get warnings over new NODEDATA
    enum.
 -- When a job is aborted send a message for any tasks that have completed.
 -- Correction to memory per CPU calculation on system with threads and
    allocating cores or sockets.
 -- Requeue batch job if it's node reboots (used to abort the job).
 -- Enlarge maximum size of srun's hostlist file.
 -- IPMI - Fix first poll to get correct consumed_energy for a step.
jette's avatar
jette committed
 -- Correction to job state recovery logic that could result in assert failure.
 -- Record partial step accounting record if allocated nodes fail abnormally.
 -- Accounting - fix issue where PrivateData=jobs or users could potentially
    show information to users that had no associations on the system.
 -- Make PrivateData in slurmdbd.conf case insensitive.
 -- sacct/sstat - Add format option ConsumedEnergyRaw to print full energy
    values.
* Changes in Slurm 2.6.0rc2
===========================
 -- HDF5 - Fix issue with Ubuntu where HDF5 development headers are
    overwritten by the parallel versions thus making it so we need handle
    both cases.
 -- ACCT_GATHER - handle suspending correctly for polling threads.
 -- Make SLURM_DISTRIBUTION env var hold both types of distribution if
    specified.
 -- Remove hardcoded /usr/local from slurm.spec.
 -- Modify slurmctld locking to improve performance under heavy load with
    very large numbers of batch job submissions or job cancellations.
 -- sstat - Fix issue where if -j wasn't given allow last argument to be checked
    for as the job/step id.
 -- IPMI - fix adjustment on poll when using EnergyIPMICalcAdjustment.
* Changes in Slurm 2.6.0rc1
===========================
 -- Added helper script for launching symmetric and MIC-only MPI tasks within
    SLURM (in contribs/mic/mpirun-mic).
 -- Change maximum delay for state save from 2 secs to 5 secs. Make timeout
    configurable at build time by defining SAVE_MAX_WAIT.
 -- Modify slurmctld data structure locking to interleave read and write
    locks rather than always favor write locks over read locks.
 -- Added sacct format option of "ALL" to print all fields.
 -- Deprecate the SchedulerParameters value of "interval" use "bf_interval"
    instead as documented.
 -- Add acct_gather_profile/hdf5 to profile jobs with hdf5
 -- Added MaxCPUsPerNode partition configuration parameter. This can be
    especially useful to schedule systems with GPUs.
 -- Permit "scontrol reboot_node" for nodes in MAINT reservation.
 -- Added "PriorityFlags" value of "SMALL_RELATIVE_TO_TIME". If set, the job's
    size component will be based upon not the job size alone, but the job's
    size divided by it's time limit.
 -- Added sbatch option "--ignore-pbs" to ignore "#PBS" options in the batch
    script.
 -- Rename slurm_step_ctx_params_t field from "mem_per_cpu" to "pn_min_memory".
    Job step now accepts memory specification in either per-cpu or per-node
    basis.
 -- Add ability to specify host repitition count in the srun hostfile (e.g.
    "host1*2" is equivalent to "host1,host1").
Danny Auble's avatar
Danny Auble committed

Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.6.0pre3
============================
 -- Add milliseconds to default log message header (both RFC 5424 and ISO 8601
    time formats). Disable milliseconds logging using the configure
    parameter "--disable-log-time-msec". Default time format changes to
    ISO 8601 (without time zone information). Specify "--enable-rfc5424time"
    to restore the time zone information.
 -- Add username (%u) to the filename pattern in the batch script.
 -- Added options for front end nodes of AllowGroups, AllowUsers, DenyGroups,
    and DenyUsers.
 -- Fix sched/backfill logic to initiate jobs with maximum time limit over the
    partition limit, but the minimum time limit permits it to start.
 -- gres/gpu - Fix for gres.conf file with multiple files on a single line
    using a slurm expression (e.g. "File=/dev/nvidia[0-1]").
 -- Replaced ipmi.conf with generic acct_gather.conf file for all acct_gather
    plugins.  For those doing development to use this follow the model set
    forth in the acct_gather_energy_ipmi plugin.
Danny Auble's avatar
Danny Auble committed
 -- Added more options to update a step's information
 -- Add DebugFlags=ThreadID which will print the thread id of the calling
    thread.
 -- CRAY - Allocate whole node (CPUs) in reservation despite what the
    user requests.  We have found any srun/aprun afterwards will work on a
    subset of resources.
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.6.0pre2
Morris Jette's avatar
Morris Jette committed
============================
 -- Do not purge inactive interactive jobs that lack a port to ping (added
    for MR+ operation).
 -- Advanced reservations with hostname and core counts now supports asymetric
    reservations (e.g. specific different core count for each node).
 -- Added slurmctld/dynalloc plugin for MapReduce+ support.
 -- Added "DynAllocPort" configuration parameter.
 -- Added partition paramter of SelectTypeParameters to override system-wide
    value.
 -- Added cr_type to partition_info data structure.
 -- Added allocated memory to node information available (within the existing
    select_nodeinfo field of the node_info_t data structure). Added Allocated
    Memory to node information displayed by sview and scontrol commands.
 -- Make sched/backfill the default scheduling plugin rather than sched/builtin
    (FIFO).
 -- Added support for a job having different priorities in different partitions.
 -- Added new SchedulerParameters configuration parameter of "bf_continue"
    which permits the backfill scheduler to continue considering jobs for
    backfill scheduling after yielding locks even if new jobs have been
    submitted. This can result in lower priority jobs from being backfill
    scheduled instead of newly arrived higher priority jobs, but will permit
    more queued jobs to be considered for backfill scheduling.
 -- Added support to purge reservation records from accounting.
 -- Cray - Add support for Basil 1.3
Morris Jette's avatar
Morris Jette committed
* Changes in SLURM 2.6.0pre1
============================
 -- Add "state" field to job step information reported by scontrol.
 -- Notify srun to retry step creation upon completion of other job steps
    rather than polling. This results in much faster throughput for job step
    execution with --exclusive option.
 -- Added "ResvEpilog" and "ResvProlog" configuration parameters to execute a
    program at the beginning and end of each reservation.
 -- Added "slurm_load_job_user" function. This is a variation of
    "slurm_load_jobs", but accepts a user ID argument, potentially resulting
    in substantial performance improvement for "squeue --user=ID"
 -- Added "slurm_load_node_single" function. This is a variation of
    "slurm_load_nodes", but accepts a node name argument, potentially resulting
    in substantial performance improvement for "sinfo --nodes=NAME".
 -- Added "HealthCheckNodeState" configuration parameter identify node states
    on which HealthCheckProgram should be executed.
 -- Remove sacct --dump --formatted-dump options which were deprecated in
    2.5.
 -- Added support for job arrays (phase 1 of effort). See "man sbatch" option
    -a/--array for details.
 -- Add new AccountStorageEnforce options of 'nojobs' and 'nosteps' which will
    allow the use of accounting features like associations, qos and limits but
    not keep track of jobs or steps in accounting.
 -- Cray - Add new cray.conf parameter of "AlpsEngine" to specify the
    communication protocol to be used for ALPS/BASIL.
 -- select/cons_res plugin: Correction to CPU allocation count logic in for
    cores without hyperthreading.
 -- Added new SelectTypeParameter value of "CR_ALLOCATE_FULL_SOCKET".
 -- Added PriorityFlags value of "TICKET_BASED" and merged priority/multifactor2
    plugin into priority/multifactor plugin.
 -- Add "KeepAliveTime" configuration parameter controlling how long sockets
    used for srun/slurmstepd communications are kept alive after disconnect.
 -- Added SLURM_SUBMIT_HOST to salloc, sbatch and srun job environment.
 -- Added SLURM_ARRAY_TASK_ID to environment of job array.
 -- Added squeue --array/-r option to optimize output for job arrays.
 -- Added "SlurmctldPlugstack" configuration parameter for generic stack of
    slurmctld daemon plugins.
 -- Removed contribs/arrayrun tool. Use native support for job arrays.
 -- Modify default installation locations for RPMs to match "make install":
    _prefix /usr/local
    _slurm_sysconfdir %{_prefix}/etc/slurm
    _mandir %{_prefix}/share/man
    _infodir %{_prefix}/share/info
 -- Add acct_gather_energy/ipmi which works off freeipmi for energy gathering
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.5.8
========================
 -- Fix for slurmctld segfault on NULL front-end reason field.
 -- Avoid gres step allocation errors when a job shrinks in size due to either
    down nodes or explicit resizing. Generated slurmctld errors of this type:
    "step_test ... gres_bit_alloc is NULL"
 -- Fix bug that would leak memory and over-write the AllowGroups field if on
    "scontrol reconfig" when AllowNodes is manually changed using scontrol.
 -- Get html/man files to install in correct places with rpms.
 -- Remove --program-prefix from spec file since it appears to be added by
    default and appeared to break other things.
 -- Updated the automake min version in autogen.sh to be correct.
 -- Select/cons_res - Correct total CPU count allocated to a job with
    --exclusive and --cpus-per-task options
 -- switch/nrt - Don't allocate network resources unless job step has 2+ nodes.
 -- select/cons_res - Avoid extraneous "oversubscribe" error messages.
 -- Reorder get config logic to avoid deadlock.
 -- Enforce QOS MaxCPUsMin limit when job submission contains no user-specified
    time limit.
 -- EpilogSlurmctld pthread is passed required arguments rather than a pointer
    to the job record, which under some conditions could be purged and result
    in an invalid memory reference.
* Changes in Slurm 2.5.7
========================
 -- Fix for linking to the select/cray plugin to not give warning about
    undefined variable.
 -- Add missing symbols to the xlator.h
 -- Avoid placing pending jobs in AdminHold state due to backfill scheduler
    interactions with advanced reservation.
 -- Accounting - make average by task not cpu.
 -- CRAY - Change logging of transient ALPS errors from error() to debug().
 -- POE - Correct logic to support poe option "-euidevice sn_all" and
    "-euidevice sn_single".
 -- Accounting - Fix minor initialization error.
 -- POE - Correct logic to support srun network instances count with POE.
 -- POE - With the srun --launch-cmd option, report proper task count when
    the --cpus-per-task option is used without the --ntasks option.
 -- POE - Fix logic binding tasks to CPUs.
 -- sview - Fix race condition where new information could of slipped past
    the node tab and we didn't notice.
 -- Accounting - Fix an invalid memory read when slurmctld sends data about
    start job to slurmdbd.
 -- If a prolog or epilog failure occurs, drain the node rather than setting it
    down and killing all of its jobs.
 -- Priority/multifactor - Avoid underflow in half-life calculation.
 -- POE - pack missing variable to allow fanout (more than 32 nodes)
 -- Prevent clearing reason field for pending jobs. This bug was introduced in
    v2.5.5 (see "Reject job at submit time ...").
 -- BGQ - Fix issue with preemption on sub-block jobs where a job would kill
    all preemptable jobs on the midplane instead of just the ones it needed to.
 -- switch/nrt - Validate dynamic window allocation size.
 -- BGQ - When --geo is requested do not impose the default conn_types.
Danny Auble's avatar
Danny Auble committed
 -- CRAY - Support CLE 4.2.0
 -- RebootNode logic - Defers (rather than forgets) reboot request with job
    running on the node within a reservation.
 -- switch/nrt - Correct network_id use logic. Correct support for user sn_all
    and sn_single options.
 -- sched/backfill - Modify logic to reduce overhead under heavy load.
 -- Fix job step allocation with --exclusive and --hostlist option.
 -- Select/cons_res - Fix bug resulting in error of "cons_res: sync loop not
    progressing, holding job #"
 -- checkpoint/blcr - Reset max_nodes from zero to NO_VAL on job restart.
 -- launch/poe - Fix for hostlist file support with repeated host names.
 -- priority/multifactor2 - Prevent possible divide by zero.
 -- srun - Don't check for executable if --test-only flag is used.
 -- energy - On a single node only use the last task for gathering energy.
    Since we don't currently track energy usage per task (only per step).
    Otherwise we get double the energy.
Danny Auble's avatar
Danny Auble committed
* Changes in Slurm 2.5.6
========================
 -- Gres fix for requeued jobs.
 -- Gres accounting - Fix regression in 2.5.5 for keeping track of gres
    requested and allocated.
Danny Auble's avatar
Danny Auble committed

Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.5.5
========================
 -- Fix for sacctmgr add qos to handle the 'flags' option.
 -- Export SLURM_ environment variables from sbatch, even if "--export"
    option does not explicitly list them.
 -- If node is in more than one partition, correct counting of allocated CPUs.
 -- If step requests more CPUs than possible in specified node count of job
    allocation then return ESLURM_TOO_MANY_REQUESTED_CPUS rather than
    ESLURM_NODES_BUSY and retrying.
 -- CRAY - Fix SLURM_TASKS_PER_NODE to be set correctly.
 -- Accounting - more checks for strings with a possible `'` in it.
 -- sreport - Fix by adding planned down time to utilization reports.
 -- Do not report an error when sstat identifies job steps terminated during
    its execution, but log using debug type message.
 -- Select/cons_res - Permit node removed from job by going down to be returned
    to service and re-used by another job.
 -- Select/cons_res - Tighter packing of job allocations on sockets.
 -- SlurmDBD - fix to allow user root along with the slurm user to register a
    cluster.
 -- Select/cons_res - Fix for support of consecutive node option.
 -- Select/cray - Modify build to enable direct use of libslurm library.
 -- Bug fixes related to job step allocation logic.
 -- Cray - Disable enforcement of MaxTasksPerNode, which is not applicable
    with launch/aprun.
 -- Accounting - When rolling up data from past usage ignore "idle" time from
    a reservation when it has the "Ignore_Jobs" flag set.  Since jobs could run
    outside of the reservation in it's nodes without this you could have
    double time.
 -- Accounting - Minor fix to avoid reuse of variable erroneously.
 -- Reject job at submit time if the node count is invalid. Previously such a
    job submitted to a DOWN partition would be queued.
 -- Purge vestigial job scripts when the slurmd cold starts or slurmstepd
    terminates abnormally.
Jason Bacon's avatar
Jason Bacon committed
 -- Add support for FreeBSD.
 -- Add sanity check for NULL cluster names trying to register.
 -- BGQ - Push action 'D' info to scontrol for admins.
 -- Reset a job's reason from PartitionDown when the partition is set up.
 -- BGQ - Handle issue where blocks would have a pending job on them and
    while it was free cnodes would go into software error and kill the job.
 -- BGQ - Fix issue where if for some reason we are freeing a block with
    a pending job on it we don't kill the job.
 -- BGQ - Fix race condition were a job could of been removed from a block
    without it still existing there.  This is extremely rare.
 -- BGQ - Fix for when a step completes in Slurm before the runjob_mux notifies
    the slurmctld there were software errors on some nodes.
 -- BGQ - Fix issue on state recover if block states are not around
    and when reading in state from DB2 we find a block that can't be created.
    You can now do a clean start to rid the bad block.
 -- Modify slurmdbd to retransmit to slurmctld daemon if it is not responding.
 -- BLUEGENE - Fix issue where when doing backfill preemptable jobs were
    never looked at to determine eligibility of backfillable job.
 -- Cray/BlueGene - Disable srun --pty option unless LaunchType=launch/slurm.
 -- CRAY - Fix sanity check for systems with more than 32 cores per node.
 -- CRAY - Remove other objects from MySQL query that are available from
    the XML.
 -- BLUEGENE - Set the geometry of a job when a block is picked and the job
    isn't a sub-block job.
 -- Cray - avoid check of macro versions of CLE for version 5.0.
 -- CRAY - Fix memory issue with reading in the cray.conf file.
 -- CRAY - If hostlist is given with srun make sure the node count is the same
    as the hosts given.
 -- CRAY - If task count specified, but no tasks-per-node, then set the tasks
    per node in the BASIL reservation request.
 -- CRAY - fix issue with --mem option not giving correct amount of memory
    per cpu.
 -- CRAY - Fix if srun --mem is given outside an allocation to set the
    APRUN_DEFAULT_MEMORY env var for aprun.  This scenario will not display
    the option when used with --launch-cmd.
 -- Change sview to use GMutex instead of GStaticMutex
 -- CRAY - set APRUN_DEFAULT_MEMROY instead of CRAY_AUTO_APRUN_OPTIONS
 -- sview - fix issue where if a partition was completely in one state the
    cpu count would be reflected correctly.
 -- BGQ - fix for handling half rack system in STATIC of OVERLAP mode to
    implicitly create full system block.
 -- CRAY - Dynamically create BASIL XML buffer to resize as needed.
 -- Fix checking if QOS limit MaxCPUMinsPJ is set along with DenyOnLimit to
    deny the job instead of holding it.
 -- Make sure on systems that use a different launcher than launch/slurm not
    to attempt to signal tasks on the frontend node.
 -- Cray - when a step is requested count other steps running on nodes in the
    allocation as taking up the entire node instead of just part of the node
    allocated.  And always enforce exclusive on a step request.
 -- Cray - display correct nodelist, node/cpu count on steps.
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 2.5.4
========================
 -- Fix bug in PrologSlurmctld use that would block job steps until node
    responds.
 -- CRAY - If a partition has MinNodes=0 and a batch job doesn't request nodes
    put the allocation to 1 instead of 0 which prevents the allocation to
    happen.
 -- Better debug when the database is down and using the --cluster option in
    the user commands.
 -- When asking for job states with sacct, default to 'now' instead of midnight
 -- Fix for handling a test-only job or immediate job that fails while being
    built.
 -- Comment out all of the logic in the job_submit/defaults plugin. The logic
    is only an example and not meant for actual use.
 -- Eliminate configuration file 4096 character line limitation.
 -- More robust logic for tree message forward
 -- BGQ - When cnodes fail in a timeout fashion correctly look up parent
    midplane.
 -- Correct sinfo "%c" (node's CPU count) output value for Bluegene systems.
 -- Backfill - Responsive improvements for systems with large numbers of jobs
    (>5000) and using the SchedulerParameters option bf_max_job_user.
 -- slurmstepd: ensure that IO redirection openings from/to files correctly
    handle interruption
 -- BGQ - Able to handle when midplanes go into Hardware::SoftwareFailure
 -- GRES - Correct tracking of specific resources used after slurmctld restart.
    Counts would previously go negative as jobs terminate and decrement from
    a base value of zero.
 -- Fix for priority/multifactor2 plugin to not assert when configured with
    --enable-debug.
 -- Select/cons_res - If the job request specified --ntasks-per-socket and the
    allocation using is cores, then pack the tasks onto the sockets up to the
    specified value.
 -- BGQ - If a cnode goes into an 'error' state and the block containing the
    cnode does not have a job running on it do not resume the block.
 -- BGQ - Handle blocks that don't free themselves in a reasonable time better.
 -- BGQ - Fix for signaling steps when allocation ends before step.
 -- Fix for backfill scheduling logic with job preemption; starts more jobs.
 -- xcgroup - remove bugs with EINTR management in write calls
 -- jobacct_gather - fix total values to not always == the max values.
 -- Fix for handling node registration messages from older versions without
    energy data.
 -- BGQ - Allow user to request full dimensional mesh.
 -- sdiag command - Correction to jobs started value reported.
 -- Prevent slurmctld assert when invalid change to reservation with running
    jobs is made.
 -- BGQ - If signal is NODE_FAIL allow forward even if job is completing
    and timeout in the runjob_mux trying to send in this situation.
 -- BGQ - More robust checking for correct node, task, and ntasks-per-node
    options in srun, and push that logic to salloc and sbatch.
 -- GRES topology bug in core selection logic fixed.
 -- Fix to handle init.d script for querying status and not return 1 on
    success.
* Changes in SLURM 2.5.3
========================
 -- Gres/gpu plugin - If no GPUs requested, set CUDA_VISIBLE_DEVICES=NoDevFiles.
    This bug was introduced in 2.5.2 for the case where a GPU count was
    configured, but without device files.
 -- task/affinity plugin - Fix bug in CPU masks for some processors.
 -- Modify sacct command to get format from SACCT_FORMAT environment variable.
 -- BGQ - Changed order of library inclusions and fixed incorrect declaration
    to compile correctly on newer compilers
 -- Fix for not building sview if glib exists on a system but not the gtk libs.
 -- BGQ - Fix for handling a job cleanup on a small block if the job has long
    since left the system.
 -- Fix race condition in job dependency logic which can result in invalid
    memory reference.
Morris Jette's avatar
Morris Jette committed
* Changes in SLURM 2.5.2
========================
 -- Fix advanced reservation recovery logic when upgrading from version 2.4.
 -- BLUEGENE - fix for QOS/Association node limits.
 -- Add missing "safe" flag from print of AccountStorageEnforce option.
 -- Fix logic to optimize GRES topology with respect to allocated CPUs.
 -- Add job_submit/all_partitions plugin to set a job's default partition
    to ALL available partitions in the cluster.
 -- Modify switch/nrt logic to permit build without libnrt.so library.
 -- Handle srun task launch failure without duplicate error messages or abort.
 -- Fix bug in QoS limits enforcement when slurmctld restarts and user not yet
    added to the QOS list.
 -- Fix issue where sjstat and sjobexitmod was installed in 2 different RPMs.
 -- Fix for job request of multiple partitions in which some partitions lack
    nodes with required features.
 -- Permit a job to use a QOS they do not have access to if an administrator
    manually set the job's QOS (previously the job would be rejected).
 -- Make more variables available to job_submit/lua plugin: slurm.MEM_PER_CPU,
    slurm.NO_VAL, etc.
 -- Fix topology/tree logic when nodes defined in slurm.conf get re-ordered.
 -- In select/cons_res, correct logic to allocate whole sockets to jobs. Work
    by Magnus Jonsson, Umea University.
 -- In select/cons_res, correct logic when job removed from only some nodes.
 -- Avoid apparent kernel bug in 2.6.32 which apparently is solved in
    at least 3.5.0.  This avoids a stack overflow when running jobs on
    more than 120k nodes.
 -- BLUEGENE - If we made a block that isn't runnable because of a overlapping
    block, destroy it correctly.
 -- Switch/nrt - Dynamically load libnrt.so from within the plugin as needed.
    This eliminates the need for libnrt.so on the head node.
 -- BLUEGENE - Fix in reservation logic that could cause abort.
* Changes in SLURM 2.5.1
========================
 -- Correction to hostlist sorting for hostnames that contain two numeric
    components and the first numeric component has various sizes (e.g.
    "rack9blade1" should come before "rack10blade1")
 -- BGQ - Only poll on initialized blocks instead of calling getBlocks on
    each block independently.
 -- Fix of task/affinity plugin logic for Power7 processors having hyper-
    threading disabled (cpu mask has gaps).
Chris Reed's avatar
Chris Reed committed
 -- Fix of job priority ordering with sched/builtin and priority/multifactor.
Chris Read's avatar
Chris Read committed
    Patch from Chris Read.
 -- CRAY - Fix for setting up the aprun for a large job (+2000 nodes).
 -- Fix for race condition related to compute node boot resulting in node being
    set down with reason of "Node <name> unexpectedly rebooted"
 -- RAPL - Fix for handling errors when opening msr files.
 -- BGQ - Fix for salloc/sbatch to do the correct allocation when asking for
    -N1 -n#.
 -- BGQ - in emulation make it so we can pretend to run large jobs (>64k nodes)
 -- BLUEGENE - Correct method to update conn_type of a job.
 -- BLUEGENE - Fix issue with preemption when needing to preempt multiple jobs
    to make one job run.
 -- Fixed issue where if an srun dies inside of an allocation abnormally it
    would of also killed the allocation.
 -- FRONTEND - fixed issue where if a systems nodes weren't defined in the
    slurm.conf with NodeAddr's signals going to a step could be handled
    incorrectly.
 -- If sched/backfill starts a job with a QOS having NO_RESERVE and not job
    time limit, start it with the partition time limit (or one year if the
    partition has no time limit) rather than NO_VAL (140 year time limit);
 -- Alter hostlist logic to allocate large grid dynamically instead of on
    stack.
 -- Change RPC version checks to support version 2.5 slurmctld with version 2.4
    slurmd daemons.
Morris Jette's avatar
Morris Jette committed
 -- Correct core reservation logic for use with select/serial plugin.
 -- Exit scontrol command on stdin EOF.
 -- Disable job --exclusive option with select/serial plugin.
* Changes in SLURM 2.5.0
========================
 -- Add DenyOnLimit flag for QOS to deny jobs at submission time if they
    request resources that reach a 'Max' limit.
 -- Permit SlurmUser or operator to change QOS of non-pending jobs (e.g.
    running jobs).
 -- BGQ - move initial poll to beginning of realtime interaction, which will
    also cause it to run if the realtime server ever goes away.
* Changes in SLURM 2.5.0-rc2
============================
 -- Modify sbcast logic to survive slurmd daemon restart while file a
    transmission is in progress.
 -- Add retry logic to munge encode/decode calls. This is needed if the munge
    deamon is under very heavy load (e.g. with 1000 slurmd daemons per compute
    node).
 -- Add launch and acct_gather_energy plugins to RPMs.
 -- Restore support for srun "--mpi=list" option.
 -- CRAY - Introduce step accounting for a Cray.
 -- Modify srun to abandon I/O 60 seconds after the last task ends. Otherwise
    an aborted slurmstepd can cause the srun process to hang indefinitely.
 -- ENERGY - RAPL - alter code to close open files (and only open them once
    where needed)
 -- If the PrologSlurmctld fails, then requeue the job an indefinite number
    of times instead of only one time.
* Changes in SLURM 2.5.0-rc1
============================
 -- Added Prolog and Epilog Guide (web page). Based upon work by Jason Sollom,
    Cray Inc. and used by permission.
 -- Restore gang scheduling functionality. Preemptor was not being scheduled.
    Fix for bugzilla #3.
Morris Jette's avatar
Morris Jette committed
 -- Add "cpu_load" to node information. Populate CPULOAD in node information
    reported to Moab cluster manager.
Morris Jette's avatar
Morris Jette committed
 -- Preempt jobs only when insufficient idle resources exist to start job,
    regardless of the node weight.
 -- Added priority/multifactor2 plugin based upon ticket distribution system.
    Work by Janne Blomqvist, Aalto University.
 -- Add SLURM_NODELIST to environment variables available to Prolog and Epilog.
 -- Permit reservations to allow or deny access by account and/or user.