RELEASE NOTES FOR SLURM VERSION 17.02
21 November 2016

IMPORTANT NOTES:
THE MAXJOBID IS NOW 67,108,863. ANY PRE-EXISTING JOBS WILL CONTINUE TO RUN BUT
NEW JOB IDS WILL BE WITHIN THE NEW MAXJOBID RANGE. Adjust your configured
MaxJobID value as needed to eliminate any confusion.

If using the slurmdbd (Slurm DataBase Daemon) you must update this first.
The 17.02 slurmdbd will work with Slurm daemons of version 15.08 and above.
You will not need to update all clusters at the same time, but it is very
important to update slurmdbd first and having it running before updating
any other clusters making use of it.  No real harm will come from updating
your systems before the slurmdbd, but they will not talk to each other
until you do.  Also at least the first time running the slurmdbd you need to
make sure your my.cnf file has innodb_buffer_pool_size equal to at least 64M.
You can accomplish this by adding the line

innodb_buffer_pool_size=64M

under the [mysqld] reference in the my.cnf file and restarting the mysqld. The
buffer pool size must be smaller than the size of the MySQL tmpdir. This is
needed when converting large tables over to the new database schema.

Slurm can be upgraded from version 15.08 or 16.05 to version 17.02 without loss
of jobs or other state information. Upgrading directly from an earlier version
of Slurm will result in loss of state information.

If using SPANK plugins that use the Slurm APIs, they should be recompiled when
upgrading Slurm to a new major release.

NOTE: If you are not using Munge, but are using the "service" scripts to
      start Slurm daemons, then you will need to remove this check from the
      etc/slurm*service scripts.

HIGHLIGHTS
==========
 -- Added infrastructure for managing workload across a federation of clusters.
    (partial functionality in version 17.02, fully operational in May 2017)
 -- In order to support federated jobs, the MaxJobID configuration parameter
    default value has been reduced from 2,147,418,112 to 67,043,328 and its
    maximum value is now 67,108,863. Upon upgrading, any pre-existing jobs that
    have a job ID above the new range will continue to run and new jobs will get
    job IDs in the new range.
 -- Add MailDomain configuration parameter to qualify email addresses.
 -- Automatically clean up task/cgroup cpuset and devices cgroups after steps
    are completed.
 -- Added burst buffer support for job arrays. Add new SchedulerParameters
    configuration parameter of bb_array_stage_cnt=# to indicate how many pending
    tasks of a job array should be made available for burst buffer resource
    allocation.
 -- Add new sacctmgr commands: "shutdown" (shutdown the server), "list stats"
    (get server statistics) "clear stats" (clear server statistics).
 -- The database index for jobs is now 64 bits.  If you happen to be close to
    4 billion jobs in your database you will want to update your slurmctld at
    the same time as your slurmdbd to prevent roll over of this variable as
    it is 32 bit previous versions of Slurm.
 -- All memory values (in MB) are now 64 bit. Previously, nodes with > 2TB of
    memory would not schedule or enforce memory limits correctly.
 -- Removed AIX, BlueGene/L and BlueGene/P support.

RPMBUILD CHANGES
================
 -- Remove separate slurm_blcr package. If Slurm is built with BLCR support,
    the files will now be part of the main Slurm packages.
 -- Replace sjstat, seff and sjobexit RPM packages with a single "contribs"
    package.

CONFIGURATION FILE CHANGES (see man appropriate man page for details)
=====================================================================
 -- Add new TRESWeights configuration parameter to NodeName line for calculating
    how busy a node is. Currently only used for federation configurations.
 -- Add new PriorityFlags option of INCR_ONLY, which prevents a job's priority
    from being decremented.

COMMAND CHANGES (see man pages for details)
===========================================
 -- Add commands to sacctmgr for managing and displaying federations.
 -- Add salloc, sbatch and srun option of --delay-boot=<time>, which will
    temporarily delay booting nodes into the desired state for a job in the
    hope of using nodes already in the proper state which will be available at
    a later time.
 -- Add salloc/sbatch/srun --spread-job option to distribute tasks over as many
    nodes as possible. This also treats the --ntasks-per-node option as a
    maximum value.
 -- Add support for sbatch --bbf option to specify a burst buffer input file.
 -- Add SLURM_ARRAY_TASK_COUNT environment variable. Total number of tasks in a
    job array (e.g. "--array=2,4,8" will set SLURM_ARRAY_TASK_COUNT=3).
 -- Modify cpu_bind and mem_bind map and mask options to accept a repetition
    count to better support large task count. For example:
    "mask_mem:0x0f*2,0xf0*2" is equivalent to "mask_mem:0x0f,0x0f,0xf0,0xf0".
 -- Add support for --mem_bind=prefer option to prefer, but not restrict memory
    use to the identified NUMA node.
 -- Add --mem_bind option of "sort" to run zonesort on KNL nodes at step start.
 -- Do not process SALLOC_HINT, SBATCH_HINT or SLURM_HINT environment variables
    if any of the following salloc, sbatch or srun command line options are
    specified: -B, --cpu_bind, --hint, --ntasks-per-core, or --threads-per-core.

OTHER CHANGES
=============
 -- Add event trigger on burst buffer errors (see strigger man page,
    --burst_buffer option).
 -- Add job AdminComment field which can only be set by a Slurm administrator.
 -- Add mechanism to constrain kernel memory allocation using cgroups. New
    cgroup.conf parameters added: ConstrainKmemSpace, MaxKmemPercent, and
    MinKmemSpace.
 -- burst_buffer/cray: Accept new jobs on backup slurmctld daemon without access
    to dw_wlm_cli command. No burst buffer actions will take place.

API CHANGES
===========

Changed members of the following structs
========================================
Memory conversion to uint64_t in: job_descriptor, job_info, node_info,
	partition_info, resource_allocation_response_msg, slurm_ctl_conf,
	slurmd_status_msg, slurm_step_ctx_params_t
Variables converted are: actual_real_mem, def_mem_per_cpu, free_mem,
	max_mem_per_cpu, mem_spec_limit, pn_min_memory, real_memory, req_mem.


Added members to the following struct definitions
=================================================
In slurmdb_cluster_cond_t: Added List federation_list
In slurmdb_cluster_rec_t: Added fed, lock, sockfd
In struct job_record: Added fed_details
In connection_arg_t: Added bool persist
In slurmctld_lock_t: Added federation
In will_run_response_msg_t: Added double sys_usage_per to report back how busy a
	cluster is.
In slurm_ctl_conf: Added mail_domain.
In slurm_msg_t: Added buffer to keep received message buffer to use for later
		      purposes.
In job_desc_msg_t: Added fed_siblings to track which clusters have sibling jobs.
In slurm_job_info_t: Added fed_origin_str, fed_siblings, fed_siblings_str to
		      display job federation information.

Added the following struct definitions
======================================
Added slurmdb_cluster_fed_t to store federation information on
	slurmdb_cluster_rec_t.
Added slurmdb_federation_cond_t for selecting federations from db.
Added slurmdb_federation_rec_t to represent federation objects.
Added job_fed_details_t for storing federated job information.
Added sib_msg_t for sending messages to siblings.

Removed members from the following struct definitions
=====================================================

Changed the following enums and #defines
========================================
Added MAX_JOB_ID (0x03FFFFFF)
Added DEBUG_FLAG_FEDR flag for federation debugging.
Added cluster_fed_states enum and defines for federation states.
Changed DEFAULT_MAX_JOB_ID from 0x7fff0000 to 0x03ff0000.
Added SELECT_NODEDATA_TRES_ALLOC_FMT_STR to select_nodedata_type.
Added SELECT_NODEDATA_TRES_ALLOC_WEIGHTED to select_nodedata_type.
Changed MEM_PER_CPU flag to 0x8000000000000000 from 0x80000000.
Added SLURM_MSG_KEEP_BUFFER msg flag to instruct slurm_receive_msg() to save the
	buffer ptr.

Added the following API's
=========================
Added slurm_load_federation() to retrieve federation info from cluster.
Added slurm_print_federation() to print federation info retrieved from cluster.
Added slurm_get_priority_flags() to retrieve priority flags from slurmctld_conf.

Changed the following API's
============================
