See Bugzilla #73 for more complete description of the problem.
Patch by Martin Perry, Bull.

If job is submitted to more than one partition, it's partition pointer can
be set to an invalid value. This can result in the count of CPUs allocated
on a node being bad, resulting in over- or under-allocation of its CPUs.
Patch by Carles Fenoy, BSC.

Hi all,

After a tough day I've finally found the problem and a solution for 2.4.1
I was able to reproduce the explained behavior by submitting jobs to 2 partitions.
This makes the job to be allocated in one partition but in the schedule function the partition of the job is changed to the NON allocated one. This makes that the resources can not be free at the end of the job.

I've solved this by changing the IS_PENDING test some lines above in the schedule function in (job_scheduler.c)

This is the code from the git HEAD (Line 801). As this file has changed a lot from 2.4.x I have not done a patch but I'm commenting the solution here.
I've moved the if(!IS_JOB_PENDING) after the 2nd line (part_ptr...). This prevents the partition of the job to be changed if it is already starting in another partition.

job_ptr = job_queue_rec->job_ptr;

part_ptr = job_queue_rec->part_ptr;
job_ptr->part_ptr = part_ptr;
xfree(job_queue_rec);

if (!IS_JOB_PENDING(job_ptr))

continue; /* started in other partition */

Hope this is enough information to solve it.

I've just realized (while writing this mail) that my solution has a memory leak as job_queue_rec is not freed.

Regards,
Carles Fenoy

there are jobs running on that hardware.

Add support for advanced reservation for specific cores rather than whole
nodes. Current limiations: homogeneous cluster, nodes idle when reservation
created, and no more than one reservation per node. Code is still under
development. Work by Alejandro Lucero Palau, et. al, BSC.

correctly when transitioning.  This also applies for 2.4.0 -> 2.4.1, no
state will be lost. (Thanks to Carles Fenoy)

Add reservation flag of Part_Nodes to allocate all nodes in a partition to
a reservation and automatically change the reservation when nodes are
added to or removed from the reservation. Based upon work by
Bill Brophy, Bull.

When running with multiple slurmd daemons per node, enable specifying a
range of ports on a single line of the node configuration in slurm.conf.
For example:
NodeName=tux[0-999] NodeAddr=localhost Port=9000-9999 ...

bg.properties in order for the runjob_mux to run correctly.

Signed-off-by: Danny Auble <da@schedmd.com>

but job is going to be canceled because it is interactive or other reason
it now receives the grace time.

check if a block is still makable if the cable wasn't in error.

removal of the job on the block failed.

29d79ef8

same time a block is destroyed and that block just happens to be the
smallest overlapping block over the bad hardware.

but not node count the node count is correctly figured out.

Without this fix, gang scheduling mode could start without creating
a list resulting in an assert when jobs are submitted.

packing the step layout structure.

we must use a small block instead of a shared midplane block.

still messages we find when we poll but haven't given it back to the real
time yet.

a job kill timeout aren't always reported to the system.  This is now
handled by the runjob_mux plugin.

sub-blocks.

to make a larger small block and are running with sub-blocks.

function didn't always work correctly.

the next step in the allocation only uses part of the allocation it gets
the correct cnodes.

In etc/init.d/slurm move check for scontrol after sourcing
/etc/sysconfig/slurm. Patch from Andy Wettstein, University of Chicago.

Change license option count separator from "*" to ":" for consistency with
the gres option (e.g. "--licenses=foo:2 --gres=gpu:2"). The "*" will still
be accepted, but is no longer documented.

maintain.

This change makes the code consistent with the documentation.
Note that "bf_res=" will continue to be recognized for now.
Patch from Rod Schultz, Bull.