RELEASE NOTES FOR SLURM VERSION 15.08
10 December 2014


IMPORTANT NOTES:
ANY JOBS WITH A JOB ID ABOVE 2,147,463,647 WILL BE PURGED WHEN SLURM IS
UPGRADED FROM AN OLDER VERSION! Reduce your configured MaxJobID value as needed
prior to upgrading in order to eliminate these jobs.

If using the slurmdbd (Slurm DataBase Daemon) you must update this first.
The 15.08 slurmdbd will work with Slurm daemons of version 14.03 and above.
You will not need to update all clusters at the same time, but it is very
important to update slurmdbd first and having it running before updating
any other clusters making use of it.  No real harm will come from updating
your systems before the slurmdbd, but they will not talk to each other
until you do.  Also at least the first time running the slurmdbd you need to
make sure your my.cnf file has innodb_buffer_pool_size equal to at least 64M.
You can accomplish this by adding the line

innodb_buffer_pool_size=64M

under the [mysqld] reference in the my.cnf file and restarting the mysqld.
This is needed when converting large tables over to the new database schema.

Slurm can be upgraded from version 14.03 or 14.11 to version 15.08 without loss
of jobs or other state information. Upgrading directly from an earlier version
of Slurm will result in loss of state information.

If using SPANK plugins, they should be recompiled against this version.


HIGHLIGHTS
==========
 -- In order to support inter-cluster job dependencies, the MaxJobID
    configuration parameter default value has been reduced from 4,294,901,760
    to 2,147,418,112 and it's maximum value is now 2,147,463,647.
    ANY JOBS WITH A JOB ID ABOVE 2,147,463,647 WILL BE PURGED WHEN SLURM IS
    UPGRADED FROM AN OLDER VERSION!
 -- Add burst buffer support infrastructure. Currently available plugin include
    burst_buffer/generic (uses administrator supplied programs to manage file
    staging) and burst_buffer/cray (uses Cray APIs to manage files).
 -- Add support for job dependencies joined with OR operator (e.g.
    "--depend=afterok:123?afternotok:124").
 -- Add advance reservation flag of "replace" that causes allocated resources
    to be replaced with idle resources. This maintains a pool of available
    resources that maintains a constant size (to the extent possible).
 -- Permit PreemptType=qos and PreemptMode=suspend,gang to be used together.
    A high-priority QOS job will now oversubscribe resources and gang schedule,
    but only if there are insufficient resources for the job to be started
    without preemption. NOTE: That with PreemptType=qos, the partition's
    Shared=FORCE:# configuration option will permit one job more per resource
    to be run than than specified, but only if started by preemption.
 -- A partition can now have a QOS attached.  This will allow a partition
    to have all the limits a QOS has.  If a limit is set in both QOS
    the partition QOS will override the job's QOS unless the job's QOS has the
    'PartitionQOS' flag set.
 -- Expanded --cpu-freq parameters to include min-max:governor specifications.
    --cpu-freq now supported on salloc and sbatch.
 -- Add support for optimized job allocations with respect to SGI Hypercube
    topology.
    NOTE: Only supported with select/linear plugin.
    NOTE: The program contribs/sgi/netloc_to_topology can be used to build
    Slurm's topology.conf file.

RPMBUILD CHANGES
================


CONFIGURATION FILE CHANGES (see man appropriate man page for details)
=====================================================================
 -- Remove DynAllocPort configuration parameter.
 -- Added new configuration parameters to support burst buffers:
    BurstBufferParameters, and BurstBufferType.
 -- Added SchedulerParameters option of "bf_busy_nodes". When selecting
    resources for pending jobs to reserve for future execution (i.e. the job
    can not be started immediately), then preferentially select nodes that are
    in use. This will tend to leave currently idle resources available for
    backfilling longer running jobs, but may result in allocations having less
    than optimal network topology. This option is currently only supported by
    the select/cons_res plugin.
 -- Added "EioTimeout" parameter to slurm.conf. It is the number of seconds srun
    waits for slurmstepd to close the TCP/IP connection used to relay data
    between the user application and srun when the user application terminates.
 -- Remove the CR_ALLOCATE_FULL_SOCKET configuration option.  It is now the
    default.
 -- Added DebugFlags values of "CpuFrequency", "Power" and "SICP".
 -- Added CpuFreqGovernors which lists governors allowed to be set with
    --cpu-freq on salloc, sbatch, and srun.
 -- Interpret a partition configuration of "Nodes=ALL" in slurm.conf as
    including all nodes defined in the cluster.
 -- Added new configuration parameters PowerParameters and PowerPlugin.
 -- Add AuthInfo option of "cred_expire=#" to specify the lifetime of a job
    step credential. The default value was changed from 1200 to 120 seconds.
    This value also controls how long a requeued job must wait before it can
    be started again.

DBD CONFIGURATION FILE CHANGES (see "man slurmdbd.conf" for details)
====================================================================


COMMAND CHANGES (see man pages for details)
===========================================
 -- Added "--cpu_freq" option to salloc and sbatch.
 -- Add sbcast support for file transfer to resources allocated to a job step
    rather than a job allocation (e.g. "sbcast -j 123.4 ...").
 -- Added new job state of STOPPED indicating processes have been stopped with a
    SIGSTOP (using scancel or sview), but retain its allocated CPUs. Job state
    returns to RUNNING when SIGCONT is sent (also using scancel or sview).
 -- The task_dist_states variable has been split into "flags" and "base"
    components. Added SLURM_DIST_PACK_NODES and SLURM_DIST_NO_PACK_NODES values
    to give user greater control over task distribution. The srun --dist options
    has been modified to accept a "Pack" and "NoPack" option. These options can
    be used to override the CR_PACK_NODE configuration option.
 -- Added BurstBuffer specification to advanced reservation.
 -- For advanced reservation, replace flag "License_only" with flag "Any_Nodes".
    It can be used to indicate the an advanced reservation resources (licenses
    and/or burst buffers) can be used with any compute nodes.
 -- Add "--sicp" (available for inter-cluster dependencies) and "--power"
    (specify power management options) to salloc, sbatch and srun commands.
 -- Added "--mail=stage_out" option to job submission commands to notify user
    when burst buffer state out is complete.
 -- Require a "Reason" when using scontrol to set a node state to DOWN.
 -- Mail notifications on job BEGIN, END and FAIL now apply to a job array as a
    whole rather than generating individual email messages for each task in the
    job array.
 -- Remove srun --max-launch-time option. The option has not been functional
    or documented since Slurm version 2.0.

OTHER CHANGES
=============


API CHANGES
===========

Changed members of the following structs
========================================
 -- Changed the following fields to struct struct job_descriptor
    cpu_freq renamed cpu_freq_max.
 -- Changed the following fields to struct job_info
    cpu_freq renamed cpu_freq_max.
 -- Changed the following fields to struct slurm_step_ctx_params_t
    cpu_freq renamed cpu_freq_max.
 -- Changed the following fields to struct slurm_step_launch_params_t
    cpu_freq renamed cpu_freq_max.
 -- Changed the following fields to struct job_step_info_t
    cpu_freq renamed cpu_freq_max.
 -- Changed the following fields to struct resource_allocation_response_msg_t
    cpu_freq renamed cpu_freq_max.


Added the following struct definitions
======================================
 -- Added the following fields to struct job_descriptor
    clusters, cpu_freq_min, cpu_freq_gov, power_flags, sicp_mode.
 -- Added the following fields to struct job_info
    cpu_freq_min, cpu_freq_gov, power_flags, sicp_mode.
 -- Added the following fields to struct slurm_step_ctx_params_t
    cpu_freq_min, cpu_freq_gov.
 -- Added the following fields to struct slurm_step_launch_params_t
    cpu_freq_min, cpu_freq_gov.
 -- Added the following fields to struct job_step_info_t
    cpu_freq_min, cpu_freq_gov.
 -- Added the following fields to struct resource_allocation_response_msg_t
    cpu_freq_min, cpu_freq_gov.
 -- Added the following fields to struct reserve_info_t:
    burst_buffer.
 -- Added the following fields to struct resv_desc_msg_t:
    burst_buffer.

Added the following struct definitions
======================================
 -- Added CPU_FREQ_GOV_MASK: mask for all defined cpu-frequency governors.


Changed the following enums and #defines
========================================
 -- Added MAIL_JOB_STAGE_OUT

Added the following API's
=========================


Changed the following API's
============================


DBD API Changes
===============

Changed members of the following structs
========================================


Added the following struct definitions
======================================

Added the following enums and #defines
========================================

Added the following API's
=========================
