NEWS 439 KB
Newer Older
    are included in the job allocation.
 -- Added ChosLoc configuration parameter to specifiy the pathname of the
    Chroot OS tool.
Morris Jette's avatar
Morris Jette committed
 -- Sent SIGCONT/SIGTERM when a job is selected for preemption with GraceTime
    configured rather than waiting for GraceTime to be reached before notifying
    the job.
 -- Do not resume a job with specialized cores on a node running another job
    with specialized cores (only one can run at a time).
 -- Add specialized core count to job suspend/resume calls.
 -- task/affinity and task/cgroup - Correct specialized core task binding with
    user supplied invalid CPU mask or map.
 -- Add srun --cpu-freq options to set the CPU governor (OnDemand, Performance,
    PowerSave or UserSpace).
 -- Add support for a job step's CPU governor and/or frequency to be reset on
    suspend/resume (or gang scheduling). The default for an idle CPU will now
    be "ondemand" rather than "userspace" with the lowest frequency (to recover
    from hard slurmd failures and support gang scheduling).
 -- Added PriorityFlags option of Calulate_Running to continue recalculating
    the priority of running jobs.
 -- Replace round-robin front-end node selection with least-loaded algorithm.
 -- CRAY - Improve support of XC30 systems when running natively.
 -- Add new node configuration parameters CoreSpecCount, CPUSpecList and
    MemSpecLimit which support the reservation of resources for system use
    with Linux cgroup.
 -- Add child_forked() function to the slurm_acct_gather_profile plugin to
    close open files, leaving application with no extra open file descriptors.
 -- Cray/ALPS system - Enable backup controller to run outside of the Cray to
    accept new job submissions and most other operations on the pending jobs.
 -- Have sacct print job and task array id's for job arrays.
Artem Polyakov's avatar
Artem Polyakov committed
 -- Smooth out fanout logic
 -- If <sys/prctl.h> is present name major threads in slurmctld, for
    example backfill
    thread: slurmctld_bckfl, the rpc manager: slurmctld_rpcmg etc.
    The name can be seen for example using top -H.
 -- sview - Better job_array support.
 -- Provide more precise error message when job allocation can not be satisfied
    (e.g. memory, disk, cpu count, etc. rather than just "node configuration
    not available").
Morris Jette's avatar
Morris Jette committed
 -- Create a new DebugFlags named TraceJobs in slurm.conf to print detailed
    information about jobs in slurmctld. The information include job ids, state
    and node count.
 -- When a job dependency can never be satisfied do not cancel the job but keep
    pending with reason WAIT_DEP_INVALID (DependencyNeverSatisfied).
* Changes in Slurm 14.03.12
===========================
 -- Make it so previous versions of salloc/srun work with newer versions
    of Slurm daemons.
Aaron Knister's avatar
Aaron Knister committed
 -- PMI2 race condition fix.
 -- Avoid delay on commit for PMI rank 0 to improve performance with some
    MPI implementations.
 -- Correct the sbatch pbs parser to process -j.
 -- Squeue modified to not merge tasks of a job array if their wait reasons
    differ.
 -- Use the slurm_getpwuid_r wrapper of getpwuid_r to handle possible
    interrupts.
 -- Allow --ignore-pbs to take effect when read as an #SBATCH argument.
 -- Do not launch step if job killed while the prolog was running.
* Changes in Slurm 14.03.11
===========================
 -- ALPS - Fix depth for Memory items in BASIL with CLE 5.2
    (changed starting in 5.2.3).
 -- ALPS - Fix issue when tracking memory on a PerNode basis instead of
    PerCPU.
 -- Modify assoc_mgr_fill_in_qos() to allow for a flag to know if the QOS read
    lock was locked outside of the function or not.
 -- Give even better estimates on pending node count if no node count
    is requested.
 -- Fix jobcomp/mysql plugin for MariaDB 10+/Mysql 5.6+ to work with reserved
    work "partition".
 -- If requested (scontrol reboot node_name) reboot a node even if it has
    an maintenance reservation that is not active yet.
 -- Fix issue where exclusive allocations wouldn't lay tasks out correctly
    with CR_PACK_NODES.
 -- Do not requeue a batch job from slurmd daemon if it is killed while in
    the process of being launched (a race condition introduced in v14.03.9).
 -- Do not let srun overwrite SLURM_JOB_NUM_NODES if already in an allocation.
 -- Prevent a job's end_time from being too small after a basil reservation
    error.
 -- Fix sbatch --ntasks-per-core option from setting invalid
    SLURM_NTASKS_PER_CORE environment value.
 -- Prevent scancel abort when no job satisfies filter options.
 -- ALPS - Fix --ntasks-per-core option on multiple nodes.
 -- Double max string that Slurm can pack from 16MB to 32MB to support
    larger MPI2 configurations.
 -- Fix Centos5 compile issues.
 -- Log Cray MPI job calling exit() without mpi_fini(), but do not treat it as
    a fatal error. This partially reverts logic added in version 14.03.9.
 -- sview - Fix displaying of suspended steps elapsed times.
 -- Increase number of messages that get cached before throwing them away
    when the DBD is down.
 -- Fix jobs from starting in overlapping reservations that won't finish before
    a "maint" reservation begins.
 -- Fix "squeue --start" to override SQUEUE_FORMAT env variable.
 -- Restore GRES functionality with select/linear plugin. It was broken in
    version  14.03.10.
 -- Fix possible race condition when attempting to use QOS on a system running
    accounting_storage/filetxt.
 -- Sanity check for Correct QOS on startup.
* Changes in Slurm 14.03.10
===========================
 -- Fix a few sacctmgr error messages.
 -- Treat non-zero SlurmSchedLogLevel without SlurmSchedLogFile as a fatal
    error.
David Bigagli's avatar
David Bigagli committed
 -- Correct sched_config.html documentation SchedulingParameters
    should be SchedulerParameters.
 -- When using gres and cgroup ConstrainDevices set correct access
    permission for the batch step.
 -- Fix minor memory leak in jobcomp/mysql on slurmctld reconfig.
 -- Fix bug that prevented preservation of a job's GRES bitmap on slurmctld
    restart or reconfigure (bug was introduced in 14.03.5 "Clear record of a
    job's gres when requeued" and only applies when GRES mapped to specific
    files).
 -- BGQ: Fix race condition when job fails due to hardware failure and is
    requeued. Previous code could result in slurmctld abort with NULL pointer.
 -- Prevent negative job array index, which could cause slurmctld to crash.
 -- Fix issue with squeue/scontrol showing correct node_cnt when only tasks
    are specified.
 -- Check the status of the database connection before using it.
 -- ALPS - If an allocation requests -n set the BASIL -N option to the
    amount of tasks / number of node.
 -- ALPS - Don't set the env var APRUN_DEFAULT_MEMORY, it is not needed anymore.
 -- Fix potential buffer overflow.
 -- Give better estimates on pending node count if no node count is requested.
 -- BLUEGENE - Fix issue where requeuing jobs could cause an assert.
Danny Auble's avatar
Danny Auble committed
* Changes in Slurm 14.03.9
==========================
 -- If slurmd fails to stat(2) the configuration print the string describing
    the error code.
 -- Fix for mixing core base reservations with whole node based reservations
    to avoid overlapping erroneously.
 -- BLUEGENE - Remove references to Base Partition.
 -- sview - If compiled on a non-bluegene system then used to view a BGQ fix
    to allow sview to display blocks correctly.
David Bigagli's avatar
David Bigagli committed
 -- Fix bug in update reservation. When modifying the reservation the end time
    was set incorrectly.
 -- The start time of a reservation that is in ACTIVE state cannot be modified.
 -- Update the cgroup documentation about release agent for devices.
 -- MYSQL - fix for setting up preempt list on a QOS for multiple QOS.
Morris Jette's avatar
Morris Jette committed
 -- Correct a minor error in the scancel.1 man page related to the
 -- Enhance the scancel.1 man page to document the sequence of signals sent
Morris Jette's avatar
Morris Jette committed
 -- Fix slurmstepd core dump if the cgroup hierarchy is not completed
 -- Fix hostlist_shift to be able to give correct node names on names with a
    different number of dimensions than the cluster.
 -- BLUEGENE - Fix invalid pointer in corner case in the plugin.
 -- Make sure on a reconfigure the select information for a node is preserved.
 -- Correct logic to support job GRES specification over 31 bits (problem
    in logic converting int to uint32_t).
 -- Remove logic that was creating GRES bitmap for node when not needed (only
    needed when GRES mapped to specific files).
 -- BLUEGENE - Fix sinfo -tr before it would only print idle nodes correctly.
 -- BLUEGENE - Fix for licenses_only reservation on bluegene systems.
 -- sview - Verify pointer before using strchr.
 -- -M option on tools talking to a Cray from a non-Cray fixed.
 -- CRAY - Fix rpmbuild issue for missing file slurm.conf.template.
 -- Fix race condition when dealing with removing many associations at
    different times when reservations are using the associations that are
    being deleted.
 -- When a node's state is set to power_down/power_up, then execute
    SuspendProgram/ResumeProgram even if previously executed for that node.
 -- Fix logic determining when job configuration (i.e. running node power up
    logic) is complete.
 -- Setting the state of a node in powered down state node to "resume" will
    no longer cause it to reboot, but only clear the "drain" state flag.
 -- Fix srun documentation to remove SLURM_NODELIST being equivalent as the -w
    option (since it isn't).
 -- Fix issue with --hint=nomultithread and allocations with steps running
    arbitrary layouts (test1.59).
 -- PrivateData=reservation modified to permit users to view the reservations
    which they have access to (rather then preventing them from seeing ANY
    reservation).  Backport from 14.11 commit 77c2bd25c.
 -- Fix PrivateData=reservation when using associations to give privileges to
    a reservation.
 -- Better checking to see if select plugin is linear or not.
 -- Add support for time specification of "fika" (3 PM).
 -- Standardize qstat wrapper more.
 -- Provide better estimate of minimum node count for pending jobs using more
    job parameters.
 -- ALPS - Add SubAllocate to cray.conf file for those who like the way <=2.5
    did the ALPS reservation.
 -- Safer check to avoid invalid reads when shutting down the slurmctld with
    lots of jobs.
 -- Fix minor memory leak in the backfill scheduler when shutting down.
 -- Add ArchiveResvs to the output of sacctmgr show config and init the variable
    on slurmdbd startup.
 -- SLURMDBD - Only set the archive flag if purging the object
    (i.e ArchiveJobs PurgeJobs).  This is only a cosmetic change.
 -- Fix for job step memory allocation logic if step requests GRES and memory
    is not allocations are not managed.
 -- Fix sinfo to display mixed nodes as allocated in '%F' output.
 -- Sview - Fix cpu and node counts for partitions.
 -- Ignore NO_VAL in SLURMDB_PURGE_* macros.
 -- ALPS - Don't drain nodes if epilog fails.  It leaves them in drain state
    with no way to get them out.
 -- Fix issue with task/affinity oversubscribing cpus erroneously when
    using --ntasks-per-node.
 -- MYSQL - Fix load of archive files.
 -- Treat Cray MPI job calling exit() without mpi_fini() as fatal error for
    that specific task and let srun handle all timeout logic.
 -- Fix small memory leak in jobcomp/mysql.
 -- Correct tracking of licenses for suspended jobs on slurmctld reconfigure or
    restart.
 -- If failed to launch a batch job requeue it in hold.
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 14.03.8
==========================
 -- Fix minor memory leak when Job doesn't have nodes on it (Meaning the job
    has finished)
 -- Fix sinfo/sview to be able to query against nodes in reserved and other
    states.
 -- Make sbatch/salloc read in (SLURM|(SBATCH|SALLOC))_HINT in order to
    handle sruns in the script that will use it.
 -- srun properly interprets a leading "." in the executable name based upon
    the working directory of the compute node rather than the submit host.
 -- Fix Lustre misspellings in hdf5 guide
 -- Fix wrong reference in slurm.conf man page to what --profile option should
    be used for AcctGatherFilesystemType.
 -- Update HDF5 document to point out the SlurmdUser is who creates the
    ProfileHDF5Dir directory as well as all it's sub-directories and files.
 -- CRAY NATIVE - Remove error message for srun's ran inside an salloc that
    had --network= specified.
Morris Jette's avatar
Morris Jette committed
 -- Defer job step initiation of required GRES are in use by other steps rather
    than immediately returning an error.
 -- Deprecate --cpu_bind from sbatch and salloc.  These never worked correctly
    and only caused confusion since the cpu_bind options mostly refer to a
    step we opted to only allow srun to set them in future versions.
 -- Modify sgather to work if Nodename and NodeHostname differ.
 -- Changed use of JobContainerPlugin where it should be JobContainerType.
 -- Fix for possible error if job has GRES, but the step explicitly requests a
    GRES count of zero.
 -- Make "srun --gres=none ..." work when executed without a job allocation.
 -- Change the global eio_shutdown_time to a field in eio handle.
Morris Jette's avatar
Morris Jette committed
 -- Advanced reservation fixes for heterogeneous systems, especially when
    reserving cores.
 -- If --hint=nomultithread is used in a job allocation make sure any srun's
    ran inside the allocation can read the environment correctly.
 -- If batchdir can't be made set errno correctly so the slurmctld is notified
    correctly.
 -- Remove repeated batch complete if batch directory isn't able to be made
    since the slurmd will send the same message.
 -- sacctmgr fix default format for list transactions.
 -- BLUEGENE - Fix backfill issue with backfilling jobs on blocks already
    reserved for higher priority jobs.
 -- When creating job arrays the job specification files for each elements
    are hard links to the first element specification files. If the controller
    fails to make the links the files are copied instead.
 -- Fix error handling for job array create failure due to inability to copy
    job files (script and environment).
 -- Added patch in the contribs directory for integrating make version 4.0 with
    Slurm and renamed the previous patch "make-3.81.slurm.patch".
 -- Don't wait for an update message from the DBD to finish before sending rc
    message back.  In slow systems with many associations this could speed
    responsiveness in sacctmgr after adding associations.
 -- Eliminate race condition in enforcement of MaxJobCount limit for job arrays.
 -- Fix anomaly allocating cores for GRES with specific device/CPU mapping.
 -- cons_res - When requesting exclusive access make sure we set the number
    of cpus in the job_resources_t structure so as nodes finish the correct
    cpu count is displayed in the user tools.
 -- If the job_submit plugin calls take longer than 1 second to run, print a
 -- Make sure transfer_s_p_options transfers all the portions of the
    s_p_options_t struct.
 -- Correct the srun man page, the SLURM_CPU_BIND_VERBOSE, SLURM_CPU_BIND_TYPE
    SLURM_CPU_BIND_LIST environment variable are set only when task/affinity
    plugin is configured.
 -- sacct - Initialize variables correctly to avoid incorrect structure
    reference.
 -- Performance adjustment to avoid calling a function multiple times when it
    only needs to be called once.
 -- Give more correct waiting reason if job is waiting on association/QOS
    MaxNode limit.
 -- DB - When sending lft updates to the slurmctld only send non-deleted lfts.
 -- BLUEGENE - Fix documentation on how to build a reservation less than
    a midplane.
 -- If Slurmctld fails to read the job environment consider it an error
 -- Add the name of the node a job is running on to the message printed by
    slurmstepd when terminating a job.
 -- Remove unsupported options from sacctmgr help and the dump function.
 -- Update sacctmgr man page removing reference to obsolete parameter
    MaxProcSecondsPerJob.
 -- Added more validity checking of incoming job submit requests.
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 14.03.7
==========================
 -- Correct typos in man pages.
 -- Add note to MaxNodesPerUser and multiple jobs running on the same node
    counting as multiple nodes.
 -- PerlAPI - fix renamed call from slurm_api_set_conf_file to
    slurm_conf_reinit.
 -- Fix gres race condition that could result in job deallocation error message.
 -- Correct NumCPUs count for jobs with --exclusive option.
 -- When creating reservation with CoreCnt, check that Slurm uses
    SelectType=select/cons_res, otherwise don't send the request to slurmctld
    and return an error.
 -- Save the state of scheduled node reboots so they will not be lost should the
    slurmctld restart.
 -- In select/cons_res plugin - Insure the node count does not exceed the task
    count.
 -- switch/nrt - Do not explicitly unload windows for a job on termination,
    only unload its table (which automatically unloads its windows).
 -- When HealthCheckNodeState is configured as IDLE don't run the
    HealthCheckProgram for nodes in any other states than IDLE.
 -- Remove all slurmctld locks prior to job_submit() being called in plugins.
    If any slurmctld data structures are read or modified, add locks directly
    in the plugin.
 -- Minor sanity check to verify the string sent in isn't NULL when using
    bit_unfmt.
 -- CRAY NATIVE - Fix issue on heavy systems to only run the NHC once per
    job/step completion.
 -- Remove unneeded step cleanup for pending steps.
 -- Fix issue where if a batch job was manually requeued the batch step
    information wasn't stored in accounting.
 -- When job is release from a requeue hold state clean up its previous
    exit code.
David Bigagli's avatar
David Bigagli committed
 -- Correct the srun man page about how the output from the user application
    is sent to srun.
 -- Increase the timeout of the main thread while waiting for the i/o thread.
    Allow up to 180 seconds for the i/o thread to complete.
 -- When using sacct -c to read the job completion data compute the correct
    job elapsed time.
 -- Perl package: Define some missing node states.
 -- When using AccountingStorageType=accounting_storage/mysql zero out the
    database index for the array elements avoiding duplicate database values.
 -- Reword the explanation of cputime and cputimeraw in the sacct man page.
 -- JobCompType allows "jobcomp/mysql" as valid name but the code used
    "job_comp/mysql" setting an incorrect default database.
 -- Try to load libslurm.so only when necessary.
 -- When nodes scheduled for reboot, set state to DOWN rather than FUTURE so
    they are still visible to sinfo. State set to IDLE after reboot completes.
 -- Apply BatchStartTimeout configuration to task launch and avoid aborting
    srun commands due to long running Prolog scripts.
 -- Fix minor memory leaks when freeing node_info_t structure.
 -- Fix various memory leaks in sview
 -- If a batch script is requeued and running steps get correct exit code/signal
    previous it was always -2.
 -- If step exitcode hasn't been set display with sacct the -2 instead
    of acting like it is a signal and exitcode.
 -- Send calculated step_rc for batch step instead of raw status as
    done for normal steps.
Morris Jette's avatar
Morris Jette committed
 -- If a job times out, set the exit code in accounting to 1 instead of the
 -- Update the acct_gather.conf.5 man page removing the reference to
    InfinibandOFEDFrequency.
 -- Fix gang scheduling for jobs submitted to multiple partitions.
 -- Enable srun to submit job to multiple partitions.
 -- Update slurm.conf man page. When Epilog or Prolog fail the node state
    is set ro DRAIN.
 -- Start a job in the highest priority partition possible, even if it requires
    preempting other jobs and delaying initiation, rather than using a lower
    priority partition. Previous logic would preempt lower priority jobs, but
    then might start the job in a lower priority partition and not use the
    resources released by the preempted jobs.
 -- Fix SelectTypeParameters=CR_PACK_NODES for srun making both job and step
    resource allocation.
 -- BGQ - Make it possible to pack multiple tasks on a core when not using
    the entire cnode.
 -- MYSQL - if unable to connect to mysqld close connection that was inited.
 -- DBD - when connecting make sure we wait MessageTimeout + 5 since the
    timeout when talking to the Database is the same timeout so a race
    condition could occur in the requesting client when receiving the response
    if the database is unresponsive.
* Changes in Slurm 14.03.6
==========================
 -- Added examples to demonstrate the use of the sacct -T option to the man
    page.
 -- Fix for regression in 14.03.5 with sacctmgr load when Parent has "'"
    around it.
 -- Update comments in sacctmgr dump header.
 -- Fix for possible abort on change in GRES configuration.
 -- CRAY - fix modules file, (backport from 14.11 commit 78fe86192b.
 -- Fix race condition which could result in requeue if batch job exit and node
    registration occur at the same time.
 -- switch/nrt - Unload job tables (in addition to windows) in user space mode.
 -- Differentiate between two identical debug messages about purging vestigial
    job scripts.
 -- If the socket used by slurmstepd to communicate with slurmd exist when
    slurmstepd attempts to create it, for example left over from a previous
    requeue or crash, delete it and recreate it.
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 14.03.5
==========================
 -- If a srun runs in an exclusive allocation and doesn't use the entire
    allocation and CR_PACK_NODES is set layout tasks appropriately.
jette's avatar
jette committed
 -- Correct Shared field in job state information seen by scontrol, sview, etc.
 -- Print Slurm error string in scontrol update job and reset the Slurm errno
    before each call to the API.
 -- Fix task/cgroup to handle -mblock:fcyclic correctly
 -- Fix for core-based advanced reservations where the distribution of cores
    across nodes is not even.
 -- Fix issue where association maxnodes wouldn't be evaluated correctly if a
    QOS had a GrpNodes set.
Morris Jette's avatar
Morris Jette committed
 -- GRES fix with multiple files defined per line in gres.conf.
 -- When a job is requeued make sure accounting marks it as such.
 -- Print the state of requeued job as REQUEUED.
 -- Fix if a job's partition was taken away from it don't allow a requeue.
 -- Make sure we lock on the conf when sending slurmd's conf to the slurmstepd.
 -- Fix issue with sacctmgr 'load' not able to gracefully handle bad formatted
    file.
 -- sched/backfill: Correct job start time estimate with advanced reservations.
 -- Error message added when in proctrack/cgroup the step freezer path isn't
    able to be destroyed for debug.
 -- Added extra index's into the database for better performance when
    deleting users.
Danny Auble's avatar
Danny Auble committed
 -- Fix issue with wckeys when tracking wckeys, but not enforcing them,
    you could get multiple '*' wckeys.
 -- Fix bug which could report to squeue the wrong partition for a running job
    that is submitted to multiple partitions.
Morris Jette's avatar
Morris Jette committed
 -- Report correct CPU count allocated to job when allocated whole node even if
    not using all CPUs.
 -- If job's constraints cannot be satisfied put it in pending state with reason
    BadConstraints and don't remove it.
 -- sched/backfill - If job started with infinite time limit, set its end_time
    one year in the future.
jette's avatar
jette committed
 -- Clear record of a job's gres when requeued.
 -- Clear QOS GrpUsedCPUs when resetting raw usage if QOS is not using any cpus.
 -- Remove log message left over from debugging.
 -- When using CR_PACK_NODES fix make --ntasks-per-node work correctly.
 -- Report correct partition associated with a step if the job is submitted to
    multiple partitions.
 -- Fix to allow removing of preemption from a QOS
 -- If the proctrack plugins fail to destroy the job container print an error
    message and avoid to loop forever, give up after 120 seconds.
 -- Make srun obey POSIX convention and increase the exit code by 128 when the
 -- Sanity check for acct_gather_energy/rapl
 -- If the proctrack plugins fail to destroy the job container print an error
    message and avoid to loop forever, give up after 120 seconds.
 -- If the sbatch command specifies the option --signal=B:signum sent the signal
    to the batch script only.
 -- If we cancel a task and we have no other exit code send the signal and
    exit code.
 -- Added note about InnoDB storage engine being used with MySQL.
 -- Set the job exit code when the job is signaled and set the log level to
    debug2() when processing an already completed job.
 -- Reset diagnostics time stamp when "sdiag --reset" is called.
 -- squeue and scontrol to report a job's "shared" value based upon partition
    options rather than reporting "unknown" if job submission does not use
    --exclusive or --shared option.
 -- task/cgroup - Fix cpuset binding for batch script.
Morris Jette's avatar
Morris Jette committed
 -- sched/backfill - Fix anomaly that could result in jobs being scheduled out
    of order.
 -- Expand pseudo-terminal size data structure field sizes from 8 to 16 bits.
 -- Set the job exit code when the job is signaled and set the log level to
Danny Auble's avatar
Danny Auble committed
    debug2() when processing an already completed job.
 -- Distinguish between two identical error messages.
 -- If using accounting_storage/mysql directly without a DBD fix issue with
    start of requeued jobs.
 -- If a job fails because of batch node failure and the job is requeued and an
    epilog complete message comes from that node do not process the batch step
    information since the job has already been requeued because the epilog
    script running isn't guaranteed in this situation.
 -- Change message to note a NO_VAL for return code could of come from node
    failure as well as interactive user.
 -- Modify test4.5 to only look at one partition instead of all of them.
David Bigagli's avatar
David Bigagli committed
 -- Fix sh5util -u to accept username different from the user that runs the
    command.
David Bigagli's avatar
David Bigagli committed
 -- Corrections to man pages:salloc.1 sbatch.1 srun.1 nonstop.conf.5
    slurm.conf.5.
 -- Restore srun --pty resize ability.
 -- Have sacctmgr dump cluster handle situations where users or such have
    special characters in their names like ':'
 -- Add more debugging for information should the job ran on wrong node
    and should there be problems accessing the state files.
Morris Jette's avatar
Morris Jette committed

Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 14.03.4
Morris Jette's avatar
Morris Jette committed
==========================
 -- Fix issue where not enforcing QOS but a partition either allows or denies
    them.
Morris Jette's avatar
Morris Jette committed
 -- CRAY - Make switch/cray default when running on a Cray natively.
 -- CRAY - Make job_container/cncu default when running on a Cray natively.
 -- Disable job time limit change if it's preemption is in progress.
 -- Correct logic to properly enforce job preemption GraceTime.
 -- Fix sinfo -R to print each down/drained node once, rather than once per
    partition.
 -- If a job has non-responding node, retry job step create rather than
    returning with DOWN node error.
 -- Support SLURM_CONF path which does not have "slurm.conf" as the file name.
 -- CRAY - make job_container/cncu default when running on a Cray natively
 -- Fix issue where batch cpuset wasn't looked at correctly in
    jobacct_gather/cgroup.
 -- Correct squeue's job node and CPU counts for requeued jobs.
 -- Correct SelectTypeParameters=CR_LLN with job selecition of specific nodes.
 -- Only if ALL of their partitions are hidden will a job be hidden by default.
 -- Run EpilogSlurmctld for a job is killed during slurmctld reconfiguration.
 -- Close window with srun if waiting for an allocation and while printing
    something you also get a signal which would produce deadlock.
 -- Add SelectTypeParameters option of CR_PACK_NODES to pack a job's tasks
    tightly on its allocated nodes rather than distributing them evenly across
    the allocated nodes.
 -- cpus-per-task support: Try to pack all CPUs of each tasks onto one socket.
    Previous logic could spread the tasks CPUs across multiple sockets.
 -- Add new distribution method fcyclic so when a task is using multiple cpus
    it can bind cyclically across sockets.
 -- task/affinity - When using --hint=nomultithread only bind to the first
    thread in a core.
 -- Make cgroup task layout (block | cyclic) method mirror that of
    task/affinity.
 -- If TaskProlog sets SLURM_PROLOG_CPU_MASK reset affinity for that task
    based on the mask given.
 -- Keep supporting 'srun -N x --pty bash' for historical reasons.
 -- If EnforcePartLimits=Yes and QOS job is using can override limits, allow
    it.
Morris Jette's avatar
Morris Jette committed
 -- Fix issues if partition allows or denies account's or QOS' and either are
    not set.
 -- If a job requests a partition and it doesn't allow a QOS or account the
    job is requesting pend unless EnforcePartLimits=Yes.  Before it would
    always kill the job at submit.
David Bigagli's avatar
David Bigagli committed
 -- Fix format output of scontrol command when printing node state.
 -- Improve the clean up of cgroup hierarchy when using the
    jobacct_gather/cgroup plugin.
 -- Added SchedulerParameters value of Ignore_NUMA.
 -- Fix issues with code when using automake 1.14.1
 -- select/cons_res plugin: Fix memory leak related to job preemption.
 -- After reconfig rebuild the job node counters only for jobs that have
    not finished yet, otherwise if requeued the job may enter an invalid
 -- Do not purge the script and environment files for completed jobs on
    slurmctld reconfiguration or restart (they might be later requeued).
 -- scontrol now accepts the option job=xxx or jobid=xxx for the requeue,
    requeuehold and release operations.
 -- task/cgroup - fix to bind batch job in the proper CPUs.
 -- Added strigger option of -N, --noheader to not print the header when
    displaying a list of triggers.
 -- Modify strigger to accept arguments to the program to execute when an
    event trigger occurs.
 -- Attempt to create duplicate event trigger now generates ESLURM_TRIGGER_DUP
    ("Duplicate event trigger").
 -- Treat special characters like %A, %s etc. literally in the file names
    when specified escaped e.g. sbatch -o /home/zebra\\%s will not expand
    %s as the stepid of the running job.
 -- CRAYALPS - Add better support for CLE 5.2 when running Slurm over ALPS.
 -- Test time when job_state file was written to detect multiple primary
    slurmctld daemons (e.g. both backup and primary are functioning as
    primary and there is a split brain problem).
 -- Fix scontrol to accept update jobid=# numtasks=#
 -- If the backup slurmctld assumes primary status, then do NOT purge any
    job state files (batch script and environment files) and do not re-use them.
    This may indicate that multiple primary slurmctld daemons are active (e.g.
    both backup and primary are functioning as primary and there is a split
    brain problem).
 -- Set correct error code when requeuing a completing/pending job
 -- When checking for if dependency of type afterany, afterok and afternotok
    don't clear the dependency if the job is completing.
 -- Cleanup the JOB_COMPLETING flag and eventually requeue the job when the
    last epilog completes, either slurmd epilog or slurmctld epilog, whichever
    comes last.
 -- When attempting to requeue a job distinguish the case in which the job is
    JOB_COMPLETING or already pending.
 -- When reconfiguring the controller don't restart the slurmctld epilog if it
    is already running.
 -- Email messages for job array events print now use the job ID using the
    format "#_# (#)" rather than just the internal job ID.
Danny Auble's avatar
Danny Auble committed
 -- Set the number of free licenses to be 0 if the global license count
    decreases and total is less than in use.
 -- Add DebugFlag of BackfillMap. Previously a DebugFlag value of Backfill
    logged information about what it was doing plus a map of expected resouce
    use in the future. Now that very verbose resource use map is only logged
    with a DebugFlag value of BackfillMap
David Bigagli's avatar
David Bigagli committed
 -- Fix slurmstepd core dump.
 -- Modify the description of -E and -S option of sacct command as point in time
    'before' or 'after' the database records are returned.
Morris Jette's avatar
Morris Jette committed
 -- Correct support for partition with Shared=YES configuration.
 -- If job requests --exclusive then do not use nodes which have any cores in an
    advanced reservation. Also prevents case where nodes can be shared by other
    jobs.
 -- For "scontrol --details show job" report the correct CPU_IDs when thre are
    multiple threads per core (we are translating a core bitmap to CPU IDs).
 -- If DebugFlags=Protocol is configured in slurm.conf print details of the
    connection, ip address and port accepted by the controller.
 -- Fix minor memory leak when reading in incomplete node data checkpoint file.
Danny Auble's avatar
Danny Auble committed
 -- Enlarge the width specifier when printing partition SHARE to display larger
    sharing values.
 -- sinfo locks added to prevent possibly duplicate record printing for
    resources in multiple partitions.
Morris Jette's avatar
Morris Jette committed

* Changes in Slurm 14.03.3-2
============================
Danny Auble's avatar
Danny Auble committed
 -- BGQ - Fix issue with uninitialized variable.
* Changes in Slurm 14.03.3
==========================
 -- Correction to default batch output file name. In version 14.03.2 was using
    "slurm_<jobid>_4294967294.out" due to error in job array logic.
 -- In slurm.spec file, replace "Requires cray-MySQL-devel-enterprise" with
    "Requires mysql-devel".
Morris Jette's avatar
Morris Jette committed
* Changes in Slurm 14.03.2
==========================
 -- Fix race condition if PrologFlags=Alloc,NoHold is used.
 -- Cray - Make NPC only limit running other NPC jobs on shared blades instead
    of limited non NPC jobs.
 -- Fix for sbatch #PBS -m (mail) option parsing.
Morris Jette's avatar
Morris Jette committed
 -- Fix job dependency bug. Jobs dependent upon multiple other jobs may start
    prematurely.
Morris Jette's avatar
Morris Jette committed
 -- Set "Reason" field for all elements of a job array on short-circuited
    scheduling for job arrays.
 -- Allow -D option of salloc/srun/sbatch to specify relative path.
 -- Added SchedulerParameter of batch_sched_delay to permit many batch jobs
    to be submitted between each scheduling attempt to reduce overhead of
    scheduling logic.
 -- Added job reason of "SchedTimeout" if the scheduler was not able to reach
    the job to attempt scheduling it.
 -- Add job's exit state and exit code to email message.
 -- scontrol hold/release accepts job name option (in addition to job ID).
 -- Handle when trying to cancel a step that hasn't started yet better.
 -- Handle Max/GrpCPU limits better
 -- Add --priority option to salloc, sbatch and srun commands.
 -- Honor partition priorities over job priorities.
 -- Fix sacct -c when using jobcomp/filetxt to read newer variables
 -- Fix segfault of sacct -c if spaces are in the variables.
 -- Release held job only with "scontrol release <jobid>" and not by resetting
    the job's priority. This is needed to support job arrays better.
 -- Correct squeue command not to merge jobs with state pending and completing
    together.
 -- Fix issue where user is requesting --acctg-freq=0 and no memory limits.
 -- Fix issue with GrpCPURunMins if a job's timelimit is altered while the job
    is running.
 -- Temporary fix for handling our typemap for the perl api with newer perl.
 -- Fix allowgroup on bad group seg fault with the controller.
 -- Handle node ranges better when dealing with accounting max node limits.

* Changes in Slurm 14.03.1-2
Morris Jette's avatar
Morris Jette committed
==========================
 -- Update configure to set correct version without having to run autogen.sh
* Changes in Slurm 14.03.1
==========================
 -- Add support for job std_in, std_out and std_err fields in Perl API.
 -- Add "Scheduling Configuration Guide" web page.
 -- BGQ - fix check for jobinfo when it is NULL
 -- Do not check cleaning on "pending" steps.
Unknown's avatar
Unknown committed
 -- task/cgroup plugin - Fix for building on older hwloc (v1.0.2).
 -- In the PMI implementation by default don't check for duplicate keys.
    Set the SLURM_PMI_KVS_DUP_KEYS if you want the code to check for
    duplicate keys.
 -- Add job submission time to squeue.
 -- Permit user root to propagate resource limits higher than the hard limit
    slurmd has on that compute node has (i.e. raise both current and maximum
    limits).
 -- Fix issue with license used count when doing an scontrol reconfig.
 -- Fix the PMI iterator to not report duplicated keys.
 -- Fix issue with sinfo when -o is used without the %P option.
 -- Rather than immediately invoking an execution of the scheduling logic on
    every event type that can enable the execution of a new job, queue its
    execution. This permits faster execution of some operations, such as
    modifying large counts of jobs, by executing the scheduling logic less
    frequently, but still in a timely fashion.
 -- If the environment variable is greater than MAX_ENV_STRLEN don't
    set it in the job env otherwise the exec() fails.
 -- Optimize scontrol hold/release logic for job arrays.
 -- Modify srun to report an exit code of zero rather than nine if some tasks
    exit with a return code of zero and others are killed with SIGKILL. Only an
    exit code of zero did this.
David Bigagli's avatar
David Bigagli committed
 -- Fix a typo in scontrol man page.
 -- Avoid slurmctld crash getting job info if detail_ptr is NULL.
 -- Fix sacctmgr add user where both defaultaccount and accounts are specified.
 -- Added SchedulerParameters option of max_sched_time to limit how long the
    main scheduling loop can execute for.
 -- Added SchedulerParameters option of sched_interval to control how frequently
    the main scheduling loop will execute.
 -- Move start time of main scheduling loop timeout after locks are aquired.
 -- Add squeue job format option of "%y" to print a job's nice value.
 -- Update scontrol update jobID logic to operate on entire job arrays.
 -- Fix PrologFlags=Alloc to run the prolog on each of the nodes in the
    allocation instead of just the first.
 -- Fix race condition if a step is starting while the slurmd is being
    restarted.
 -- Make sure a job's prolog has ran before starting a step.
 -- BGQ - Fix invalid memory read when using DefaultConnType in the
    bluegene.conf
 -- Make sure we send node state to the DBD on clean start of controller.
 -- Fix some sinfo and squeue sorting anomalies due to differences in data
    types.
 -- Only send message back to slurmctld when PrologFlags=Alloc is used on a
    Cray/ALPS system, otherwise use the slurmd to wait on the prolog to gate
    the start of the step.
 -- Remove need to check PrologFlags=Alloc in slurmd since we can tell if prolog
    has ran yet or not.
 -- Fix squeue to use a correct macro to check job state.
 -- BGQ - Fix incorrect logic issues if MaxBlockInError=0 in the bluegene.conf.
 -- priority/basic - Insure job priorities continue to decrease when jobs are
    submitted with the --nice option.
 -- Make the PrologFlag=Alloc work on batch scripts
 -- Make PrologFlag=NoHold (automatically sets PrologFlag=Alloc) not hold in
    salloc/srun, instead wait in the slurmd when a step hits a node and the
    prolog is still running.
 -- Added --cpu-freq=highm1 (high minus one) option.
 -- Expand StdIn/Out/Err string length output by "scontrol show job" from 128
    to 1024 bytes.
 -- squeue %F format will now print the job ID for non-array jobs.
 -- Use quicksort for all priority based job sorting, which improves performance
    significantly with large job counts.
 -- If a job has already been released from a held state ignore successive
    release requests.
Danny Auble's avatar
Danny Auble committed
 -- Fix srun/salloc/sbatch man pages for the --no-kill option.
 -- Add squeue -L/--licenses option to filter jobs by license names.
 -- Handle abort job on node on front end systems without core dumping.
 -- Fix dependency support for job arrays.
 -- When updating jobs verify the update request is not identical to
    the current settings.
 -- When sorting jobs and priorities are equal sort by job_id.
 -- Do not overwrite existing reason for node being down or drained.
 -- Requeue batch job if Munge is down and credential can not be created.
Morris Jette's avatar
Morris Jette committed
 -- Make _slurm_init_msg_engine() tolerate bug in bind() returning a busy
Morris Jette's avatar
Morris Jette committed
 -- Don't block scheduling of entire job array if it could run in multiple
    partitions.
 -- Introduce a new debug flag Protocol to print protocol requests received
    together with the remote IP address and port.
 -- CRAY - Set up the network even when only using 1 node.
 -- CRAY - Greatly reduce the number of error messages produced from the task
    plugin and provide more information in the message.
* Changes in Slurm 14.03.0
==========================
 -- job_submit/lua: Fix invalid memory reference if script returns error message
    for user.
Morris Jette's avatar
Morris Jette committed
 -- Add logic to sleep and retry if slurm.conf can't be read.
 -- Reset a node's CpuLoad value at least once each SlurmdTimeout seconds.
 -- Scheduler enhancements for reservations: When a job needs to run in
    reservation, but can not due to busy resources, then do not block all jobs
    in that partition from being scheduled, but only the jobs in that
    reservation.
 -- Export "SLURM*" environment variables from sbatch even if --export=NONE.
 -- When recovering node state if the Slurm version is 2.6 or 2.5 set the
    protocol version to be SLURM_2_5_PROTOCOL_VERSION which is the minimum
    supported version.
 -- Update the scancel man page documenting the -s option.
 -- Update sacctmgr man page documenting how to modify account's QOS.
David Bigagli's avatar
David Bigagli committed
 -- Fix for sjstat which currently does not print >1TB memory values correctly.
 -- Change xmalloc()/xfree() to malloc()/free() in hostlist.c for better
    performance.
 -- Update squeue.1 man page describing the SPECIAL_EXIT state.
 -- Added scontrol option of errnumstr to return error message given a slurm
    error number.
 -- If srun invoked with the --multi-prog option, but no task count, then use
    the task count provided in the MPMD configuration file.
 -- Prevent sview abort on some systems when adding or removing columns to the
    display for nodes, jobs, partitions, etc.
Morris Jette's avatar
Morris Jette committed
 -- Add job array hash table for improved performance.
 -- Make AccountingStorageEnforce=all not include nojobs or nosteps.
 -- Added sacctmgr mod qos set RawUsage=0.
 -- Modify hostlist functions to accept more than two numeric ranges (e.g.
    "row[1-3]rack[0-8]slot[0-63]")
* Changes in Slurm 14.03.0rc1
Morris Jette's avatar
Morris Jette committed
==============================
 -- Fixed typos in srun_cr man page.
 -- Run job scheduling logic immediately when nodes enter service.
 -- Added sbatch '--parsable' option to output only the job id number and the
    cluster name separated by a semicolon. Errors will still be displayed.
Morris Jette's avatar
Morris Jette committed
 -- Added failure management "slurmctld/nonstop" plugin.
 -- Prevent jobs being killed when a checkpoint plugin is enabled or disabled.
 -- Update the documentation about SLURM_PMI_KVS_NO_DUP_KEYS environment
    variable.
 -- select/cons_res bug fix for range of node counts with --cpus-per-task
    option (e.g. "srun -N2-3 -c2 hostname" would allocate 2 CPUs on the first
    node and 0 CPUs on the second node).
 -- Change reservation flags field from 16 to 32-bits.
 -- Add reservation flag value of "FIRST_CORES".
 -- Added the idea of Resources to the database.  Framework for handling
    license servers outside of Slurm.
 -- When starting the slurmctld only send past job/node state information to
    accounting if running for the first time (should speed up startup
    dramatically on systems with lots of nodes or lots of jobs).
 -- Compile and run on FreeBSD 8.4.
 -- Make job array expressions more flexible to accept multiple step counts in
    the expression (e.g. "--array=1-10:2,50-60:5,123").
 -- switch/cray - add state save/restore logic tracking allocated ports.
 -- SchedulerParameters - Replace max_job_bf with bf_max_job_start (both will
    work for now).
 -- Add SchedulerParameters options of preempt_reorder_count and
    preempt_strict_order.
 -- Make memory types in acct_gather uint64_t to handle systems with more than
    4TB of memory on them.
 -- BGQ - --export=NONE option for srun to make it so only the SLURM_JOB_ID
    and SLURM_STEP_ID env vars are set.
 -- Munge plugins - Add sleep between retries if can't connect to socket.
 -- Added DebugFlags value of "License".
 -- Added --enable-developer which will give you -Werror when compiling.
 -- Fix for job request with GRES count of zero.
David Bigagli's avatar
David Bigagli committed
 -- Fix a potential memory leak in hostlist.
 -- Job array dependency logic: Cache results for major performance improvement.
 -- Modify squeue to support filter on job states Special_Exit and Resizing.
 -- Defer purging job record until after EpilogSlurmctld completes.
 -- Add -j option for jobid to sbcast.
 -- Fix handling RPCs from a 14.03 slurmctld to a 2.6 slurmd
* Changes in Slurm 14.03.0pre6
==============================
 -- Modify slurmstepd to log messages according to the LogTimeFormat
    parameter in slurm.conf.
 -- Insure that overlapping reservations do not oversubscribe available
    licenses.
 -- Added core specialization logic to select/cons_res plugin.
 -- Added whole_node field to job_resources structure and enable gang scheduling
    for jobs with core specialization.
 -- When using FastSchedule = 1 the nodes with less than configured resources
    are not longer set DOWN, they are set to DRAIN instead.
 -- Modified 'sacctmgr show associations' command to show GrpCPURunMins
    by default.
 -- Replace the hostlist_push() function with a more efficient
    hostlist_push_host().
 -- Modify the reading of lustre file system statistics to print more
    information when debug and when io error occur.
 -- Add specialized core count field to job credential data.
    NOTE: This changes the communications protocol from other pre-releases of
    version 14.03. All programs must be cancelled and daemons upgraded from
    previous pre-releases of version 14.03. Upgrades from version 2.6 or earlier
    can take place without loss of jobs
 -- Add version number to node and front-end configuration information visible
    using the scontrol tool.
 -- Add idea of a RESERVED flag for node state so idle resources are marked
    not "idle" when in a reservation.
 -- Added core specialization plugin infrastructure.
jette's avatar
jette committed
 -- Added new job_submit/trottle plugin to control the rate at which a user
    can submit jobs.
 -- CRAY - added network performance counters option.
 -- Allow scontrol suspend/resume to accept jobid in the format jobid_taskid
    to suspend/resume array elements.
Morris Jette's avatar
Morris Jette committed
 -- In the slurmctld job record, split "shared" variable into "share_res" (share
    resource) and "whole_node" fields.
 -- Fix the format of SLURM_STEP_RESV_PORTS. It was generated incorrectly
    when using the hostlist_push_host function and input surrounded by [].
 -- Modify the srun --slurmd-debug option to accept debug string tags
    (quiet, fatal, error, info verbose) beside the numerical values.
 -- Fix the bug where --cpu_bind=map_cpu is interpreted as mask_cpu.
 -- Update the documentation egarding the state of cpu frequencies after
    a step using --cpu-freq completes.
Danny Auble's avatar
Danny Auble committed
 -- CRAY - Fix issue when a job is requeued and nhc is still running as it is
    being scheduled to run again.  This would erase the previous job info
    that was still needed to clean up the nodes from the previous job run.
    (Bug 526).
 -- Set SLURM_JOB_PARTITION environment variable set for all job allocations.
 -- Set SLURM_JOB_PARTITION environment variable for Prolog program.
 -- Added SchedulerParameters option of partition_job_depth to limit scheduling
    logic depth by partition.
 -- Handle the case in which errno is not reset to 0 after calling
    getgrent_r(), which causes the controller to core dump.
Danny Auble's avatar
Danny Auble committed
* Changes in Slurm 14.03.0pre5
==============================
 -- Added squeue format option of "%X" (core specialization count).
 -- Added core specialization web page (just a start for now).
David Bigagli's avatar
David Bigagli committed
 -- Added the SLURM_ARRAY_JOB_ID and SLURM_ARRAY_TASK_ID
jette's avatar
jette committed
    in epilog slurmctld environment.
 -- Fix bug in job step allocation failing due to memory limit.
 -- Modify the pbsnodes script to reflect its output on a TORQUE system.
 -- Add ability to clear a node's DRAIN flag using scontrol or sview by setting
    it's state to "UNDRAIN". The node's base state (e.g. "DOWN" or "IDLE") will
    not be changed.
 -- Modify the output of 'scontrol show partition' by displaying
    DefMemPerCPU=UNLIMITED and MaxMemPerCPU=UNLIMITED when these limits are
    configured as 0.
 -- mpirun-mic - Major re-write of the command wrapper for Xeon Phi use.
 -- Add new configuration parameter of AuthInfo to specify port used by
    authentication plugin.
 -- Fixed conditional RPM compiling.
jette's avatar
jette committed
 -- Corrected slurmstepd ident name when logging to syslog.
 -- Fixed sh5util loop when there are no node-step files.
 -- Add SLURM_CLUSTER_NAME to environment variables passed to PrologSlurmctld,
    Prolog, EpilogSlurmctld, and Epilog
 -- Add the idea of running a prolog right when an allocation happens
    instead of when running on the node for the first time.
 -- If user runs 'scontrol reconfig' but hostnames or the host count changes
    the slurmctld throws a fatal error.
 -- gres.conf - Add "NodeName" specification so that a single gres.conf file
    can be used for a heterogeneous cluster.
 -- Add flag to accounting RPC to indicate if job data is packed or not.
 -- After all srun tasks have terminated on a node close the stdout/stderr
    channel with the slurmstepd on that node.
 -- In case of i/o error with slurmstepd log an error message and abort the
    job.
 -- Add --test-only option to sbatch command to validate the script and options.
    The response includes expected start time and resources to be allocated.
Danny Auble's avatar
Danny Auble committed
* Changes in Slurm 14.03.0pre4
==============================
 -- Remove the ThreadID documentation from slurm.conf. This functionality has
    been obsoleted by the LogTimeFormat.
 -- Sched plugins - rename global and plugin functions names for consistency
    with other plugin types.
 -- BGQ - Added RebootQOSList option to bluegene.conf to allow an implicate
    reboot of a block if only jobs in the list are running on it when cnodes
    go into a failure state.
 -- Correct task count of pending job steps.
 -- Improve limit enforcement for jobs, set RLIMIT_RSS, RLIMIT_AS and/or
    RLIMIT_DATA to enforce memory limit.
 -- Pending job steps will have step_id of INFINITE rather than NO_VAL and
    will be reported as "TBD" by scontrol and squeue commands.
 -- Add logic so PMI_Abort or PMI2_Abort can propagate an exit code.
 -- Added SlurmdPlugstack configuration parameter.
 -- Added PriorityFlag DEPTH_OBLIVIOUS to have the depth of an association
    not effect it's priorty.
 -- Multi-thread the sinfo command (one thread per partition).
Matthias Jurenz's avatar
Matthias Jurenz committed
 -- Added sgather tool to gather files from a job's compute nodes into a
    central location.
 -- Added configuration parameter FairShareDampeningFactor to offer a greater
    priority range based upon utilization.
 -- Change MaxArraySize and job's array_task_id from 16-bit to 32-bit field.
Morris Jette's avatar
Morris Jette committed
    Additional Slurm enhancements are be required to support larger job arrays.
 -- Added -S/--core-spec option to salloc, sbatch and srun commands to reserve
    specialized cores for system use. Modify scontrol and sview to get/set
Morris Jette's avatar
Morris Jette committed
    the new field. No enforcement exists yet for these new options.
    struct job_info / slurm_job_info_t: Added core_spec
    struct job_descriptorjob_desc_msg_t: Added core_spec
Danny Auble's avatar
Danny Auble committed
* Changes in Slurm 14.03.0pre3
==============================
 -- Do not set SLURM_NODEID environment variable on front-end systems.
 -- Convert bitmap functions to use int32_t instead of int in data structures
    and function arguments. This is to reliably enable use of bitmaps containing
    up to 4 billion elements. Several data structures containing index values
    were also changed from data type int to int32_t:
    - Struct job_info / slurm_job_info_t: Changed exc_node_inx, node_inx, and
      req_node_inx from type int to type int32_t
    - job_step_info_t: Changed node_inx from type int to type int32_t
    - Struct partition_info / partition_info_t: Changed node_inx from type int
      to type int32_t
    - block_job_info_t: Changed cnode_inx from type int to type int32_t
    - block_info_t: Changed ionode_inx and mp_inx from type int to type int32_t
    - Struct reserve_info / reserve_info_t: Changed node_inx from type int to
      type int32_t
 -- Modify qsub wrapper output to match torque command output, just print the
    job ID rather than "Submitted batch job #"
 -- Change Slurm error string for ESLURM_MISSING_TIME_LIMIT from
    "Missing time limit" to
    "Time limit specification required, but not provided"
 -- Change salloc job_allocate error message header from
    "Failed to allocate resources" to
    "Job submit/allocate failed"
 -- Modify slurmctld message retry logic to support Cray cold-standby SDB.
Danny Auble's avatar
Danny Auble committed
* Changes in Slurm 14.03.0pre2
==============================
 -- Added "JobAcctGatherParams" configuration parameter. Value of "NoShare"
    disables accounting for shared memory.
 -- Added fields to "scontrol show job" output: boards_per_node,
    sockets_per_board, ntasks_per_node, ntasks_per_board, ntasks_per_socket,
    ntasks_per_core, and nice.
 -- Add squeue output format options for job command and working directory
    (%o and %Z respectively).
 -- Add stdin/out/err to sview job output.
 -- Add new job_state of JOB_BOOT_FAIL for job terminations due to failure to
    boot it's allocated nodes or BlueGene block.
 -- CRAY - Add SelectTypeParameters NHC_NO_STEPS and NHC_NO which will disable
    the node health check script for steps and allocations respectfully.
 -- Reservation with CoreCnt: Avoid possible invalid memory reference.
 -- Add new error code for attempt to create a reservation with duplicate name.
 -- Validate that a hostlist file contains text (i.e. not a binary).
 -- switch/generic - propagate switch information from srun down to slurmd and
    slurmstepd.
 -- CRAY - Do not package Slurm's libpmi or libpmi2 libraries. The Cray version
    of those libraries must be used.
 -- Added a new option to the scontrol command to view licenses that are
    configured in use and avalable. 'scontrol show licenses'.
 -- MySQL - Made Slurm compatible with 5.6
Danny Auble's avatar
Danny Auble committed
* Changes in Slurm 14.03.0pre1
 -- sview - improve scalability
 -- Add task pointer to the task_post_term() function in task plugins. The
    terminating task's PID is available in task->pid.
 -- Move select/cray to select/alps
 -- Defer sending SIGKILL signal to processes while core dump in progress.
 -- Added JobContainerPlugin configuration parameter and plugin infrastructure.
 -- Added partition configuration parameters AllowAccounts, AllowQOS,
 -- The rpmbuild option for a cray system with ALPS has changed from
    %_with_cray to %_with_cray_alps.
 -- The log file timestamp format can now be selected at runtime via the
    LogTimeFormat configuration option. See the slurm.conf and slurmdbd.conf
 -- Added switch/generic plugin to a job's convey network topology.
 -- BLUEGENE - If block is in 'D' state or has more cnodes in error than
    MaxBlockInError set the job wait reason appropriately.
 -- API use: Generate an error return rather than fatal error and exit if the
    configuraiton file is absent or invalid. This will permit Slurm APIs to be
    more reliably used by other programs.
 -- Add support for load-based scheduling, allocate jobs to nodes with the
    largest number of available CPUs. Added SchedulingParameters paramter of
    "CR_LLN" and partition parameter of "LLN=yes|no".
 -- Added job_info() and step_info() functions to the gres plugins to extract
    plugin specific fields from the job's or step's GRES data structure.
 -- Added sbatch --signal option of "B:" to signal the batch shell rather than
    only the spawned job steps.
 -- Added sinfo and squeue format option of "%all" to print all fields available
    for the data type with a vertical bar separating each field.
 -- Add mechanism for job_submit plugin to generate error message for srun,
Morris Jette's avatar
Morris Jette committed
    salloc or sbatch to stderr. New argument added to job_submit function in
 -- Add StdIn, StdOut, and StdErr paths to job information dumped with
    "scontrol show job".