plugins.  For those doing development to use this follow the model set
forth in the acct_gather_energy_ipmi plugin.

We're in the process of setting up a few GPU nodes in our cluster, and
want to use Gres to control access to them.

Currently, we have activated one node with 2 GPUs.  The gres.conf file
on that node reads

----------------

Name=gpu Count=2 File=/dev/nvidia[0-1]
Name=localtmp Count=1800
----------------

(the localtmp is just counting access to local tmp disk.)  Nodes without
GPUs have gres.conf files like this:

----------------

Name=gpu Count=0
Name=localtmp Count=90
----------------

slurm.conf contains the following:

GresTypes=gpu,localtmp
Nodename=DEFAULT Sockets=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=62976 Gres=localtmp:90 State=unknown
[...]
Nodename=c19-[1-16] NodeHostname=compute-19-[1-16] Weight=15848 CoresPerSocket=4 Gres=localtmp:1800,gpu:2 Feature=rack19,intel,ib

Submitting a job with sbatch --gres:1 ... sets the CUDA_VISIBLE_DEVICES for
the job.  However, the values seem a bit strange:

- If we submit one job with --gres:1, CUDA_VISIBLE_DEVICES gets the value 0.

- If we submit two jobs with --gres:1 at the same time,
  CUDA_VISIBLE_DEVICES gets the value 0 for one job, and 1633906540 for
  the other.

- If we submit one job with --gres:2, CUDA_VISIBLE_DEVICES gets the
  value 0,1633906540

APRUN_DEFAULT_MEMORY env var for aprun.  This scenario will not display
the option when used with --launch-cmd.

per cpu.

per cpu.

If task count specified, but no tasks-per-node, then set the tasks
per node in the BASIL reservation request.

as the hosts given.

isn't a sub-block job.

the XML.

Fix for bug 258

Fix sched/backfill logic to initiate jobs with maximum time limit over the
partition limit, but the minimum time limit permits it to start.
Related to bug 251

never looked at to determine eligibility of backfillable job.

A fix for this problem will require more study. This one causes
xassert when an attempt to start a job results in it not being
started by sched/backfill due to the partition time limit.

and when reading in state from DB2 we find a block that can't be created.
You can now do a clean start to rid the bad block.

the slurmctld there were software errors on some nodes.

without it still existing there.  This is extremely rare.

a pending job on it we don't kill the job.

while it was free cnodes would go into software error and kill the job.

Fix sched/backfill logic to initiate jobs with maximum time limit over the
partition limit, but the minimum time limit permits it to start.
Related to bug 251

Fix for bug 224

WIthout this patch, when the slurmd cold starts or slurmstepd
terminates abnormally, the job script file can be left around.
bug 243

Previously such a job submitted to a DOWN partition would be queued.
bug 187

a reservation when it has the "Ignore_Jobs" flag set.  Since jobs could run
outside of the reservation in it's nodes without this you could have
double time.

This is not applicable with launch/aprun

These changes are required so that select/cray can load select/linear,
  which is a bit more complex than the other select plugin structures.
Export plugin_context_create and plugin_context_destroy symbols from
  libslurm.so.
Correct typo in exported hostlist_sort symbol name
Define some functions in select/cray to avoid undefined symbols if
  the plugin is loaded via libslurm rather than from a slurm command
  (which has all of the required symbols)