Sometimes, generally with several jobs on the same node or calling many sstat for the job, the pipe is not ready to be read.
In this case, the function reading the pipe return an error and the values of consumed energy are set to NO_VAL.
From this point, the values are never read again because the process "knows" there is no value to read.
Thus, if there is one error, NO_VAL is saved in database and no information of consumed energy is stored.

To avoid this, we wrote the attached patch.

For first read of pipe, if the pipe doesn't exist, the function retry "NBFIRSTREAD = 3" times with a waiting time of 1 second.
Then during job run and for final read, if the pipe doesn't exist, the values are not updated.

The first time, the pipe is read if the writer thread is running.
If sstat fails to read pipe, the value is not update and last value is printed.
But if there is a problem during last read:

    if there was sstat calls, the value exists but we miss all change between last sstat and end of step.
    if not, the value is just "0" (no update from the begin).

plugins.  For those doing development to use this follow the model set
forth in the acct_gather_energy_ipmi plugin.

We're in the process of setting up a few GPU nodes in our cluster, and
want to use Gres to control access to them.

Currently, we have activated one node with 2 GPUs.  The gres.conf file
on that node reads

----------------

Name=gpu Count=2 File=/dev/nvidia[0-1]
Name=localtmp Count=1800
----------------

(the localtmp is just counting access to local tmp disk.)  Nodes without
GPUs have gres.conf files like this:

----------------

Name=gpu Count=0
Name=localtmp Count=90
----------------

slurm.conf contains the following:

GresTypes=gpu,localtmp
Nodename=DEFAULT Sockets=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=62976 Gres=localtmp:90 State=unknown
[...]
Nodename=c19-[1-16] NodeHostname=compute-19-[1-16] Weight=15848 CoresPerSocket=4 Gres=localtmp:1800,gpu:2 Feature=rack19,intel,ib

Submitting a job with sbatch --gres:1 ... sets the CUDA_VISIBLE_DEVICES for
the job.  However, the values seem a bit strange:

- If we submit one job with --gres:1, CUDA_VISIBLE_DEVICES gets the value 0.

- If we submit two jobs with --gres:1 at the same time,
  CUDA_VISIBLE_DEVICES gets the value 0 for one job, and 1633906540 for
  the other.

- If we submit one job with --gres:2, CUDA_VISIBLE_DEVICES gets the
  value 0,1633906540

Conflicts:
	NEWS

APRUN_DEFAULT_MEMORY env var for aprun.  This scenario will not display
the option when used with --launch-cmd.

per cpu.

per cpu.

If task count specified, but no tasks-per-node, then set the tasks
per node in the BASIL reservation request.

Conflicts:
	src/plugins/select/cray/cray_config.c

as the hosts given.

isn't a sub-block job.

the XML.

Fix for bug 258

FrontEndNode: allow/deny groups/users
Node: Power consumption

(at initiating pending job steps), interupt driven rather than
retry based.\

Fix sched/backfill logic to initiate jobs with maximum time limit over the
partition limit, but the minimum time limit permits it to start.
Related to bug 251

Conflicts:
	src/slurmctld/job_mgr.c

This is the only remaining use of the function.
Also added boards parameter to the function and removed some unused parameters

Mostly to split long lines

    I am sending the latest update of my NPPCU-support patch for Slurm 2.5.0.  As before, this patch is applied over my basic BASIL 1.3 support patch.  The reason for this latest version is that it came to my attention, that certain jobs that should have been rejected by Slurm were allowed through.  I then further noticed that this would cause the backfill algorithm to slow down dramatically (often not being able to process any other jobs).
     The cause of the problem was that when I introduced the functionality into Slurm to properly set the "nppcu" (number of processors per compute unit) attribute in the XML reservation request to ALPS, I didn't also adjust the tests earlier in the code that eliminate nodes from consideration that do not have sufficient resources.  In other words, jobs that would exceed the absolute total number of processors on the node would be rejected as always (this is good).  Jobs that required the reduced number of "visible" processors on the node or less were allocated and worked fine (this is good).  Unfortunately, jobs that needed a number of processors somewhere in between these limits (let's call them the soft and hard limits) were allowed through by Slurm.  Making matters worse, when Slurm would subsequently try to request the ALPS reservation, ALPS would correctly reject it but Slurm would keep trying--this would then kill the backfilling.  In my opinion, these jobs should have been rejected from the onset by Slurm as they are asking for more processors per node than can be supplied.  If the user wants this number of processors they should specify the "--ntasks-per-core=..." (in our case "2" as that is the full number of hardware threads per core).  Obviously, this problem only appeared when I used CR_ONE_TASK_PER_CORE in the slurm.conf as I had modified the code to set nppcu to 1 when Slurm was configured with that option and the user didn't explicitly specify a different value.
    The patch appears to be working well for us now and so I am submitting it to you for your review.