here is information about this patch and the reasons for it: http://tech.ryancox.net/2015/04/caller-id-handling-ssh-launched-processes-in-slurm.html)

As discussed previously, here is a patch against master (branched this morning at 709f6504).  It works though I'm sure it has some rough edges that you'll find.  I had to export a few symbols that weren't there from stepd_api.[ch].  A lot of the code I had to modify is new territory for me so it's likely I made many mistakes.

There are a few minor things I might end up wanting to change (I'm not exactly in love with some of the variable or function names I chose though I can live with them).  I might make a few minor tweaks to the pam module as well but it won't affect the RPC code.  Currently the README is written like a manpage.  I might turn it into a man page and say "read the manpage" in the README.

Here is an excerpt from the README that states how decisions are made:
  1) Check the local stepds for a count of jobs owned by the non-root user
    a) If none, deny (option action_no_jobs)
    b) If only one, adopt the process into that job
    c) If multiple, continue
  2) Determine src/dst IP/port of socket
  3) Issue callerid RPC to slurmd at IP address of source
    a) If the remote slurmd can identify the source job, adopt into that job
    b) If not, continue
  4) Pick a random local job from the user to adopt into (option action_unknown)

I tried to document to thoroughly document the code, so hopefully it makes sense.  Also, I noticed that one of the stepd functions returns a uid_t which is set to -1 on error.  The problem with that is that Linux's uid_t is uint32_t.

One area of concern in the code is the stepd calls in pam_slurm_adopt.c code.  I hope I'm doing enough error handling there, but maybe not.  What happens if a step is completing or if the step data is still around even though it's actually dead?

The code to actually adopt processes is currently a no-op.  That will depend on having the allocation step code added.  I haven't checked yet to see if all the relevant plugins (proctrack, jobacct_gather, etc.) have hooks to add a new process to the plugin.  If not, it will have to be added as well.

Lastly, I exceed 80 characters on lines with user-visible strings since Slurm follows the Linux kernel coding style.  Chapter 2 of https://www.kernel.org/doc/Documentation/CodingStyle says "never break user-visible strings... because that breaks the ability to grep for them" (which I have wished Slurm followed, by the way, since I have hit that issue).  I know in the past that you wanted even those lines to be wrapped but I figured I would ask if anything has changed :)

The "flags" option was not being forwarded to the slurmctld, but
was always set to 0.

Change the scancel command to always use the job_id string based API.
Add retry logic on the job_id string logic.
Add more checking for error codes and used appropriate exit codes.
Use NO_VAL rather than "-1" for unset signal value.

Change the temporary file names used by two tests to include the
test ID number, so we can see where they came from if left around.

The jobcomp/elasticsearch plugin's Makefile.am file was accidentally
not added to GIT...

In the course of testing some scancel changes, a bunch of tests generated
"FAILURE" messages due to job cancellation failures, but the tests reported
"SUCCESS" at the end and an exit code of zero. This patch adds a checks
for the return value of the "cancel_job" procedure.

In slurmctld communication agent, make the thread timeout be the configured
value of MessageTimeout (or 30 seconds, whichever is larger) rather than
30 seconds.

Conflicts:
	src/scancel/scancel.c
	src/scancel/scancel.h

Fix scancel bug which could return an error on attempt to signal a job step.
A simple "scancel 12.3" to signal a specific job step would fail. Adding
another option (say "-i", "--partion=", etc.) would fix this.

check for more error conditions and avoid a memory leak if an error occurs.

Log when job state information is not being archived, but being cached. Print an error after each 100 jobs.

The plugin version number contents have changed in Slurm v15.08.

Recent scancel mods resulted in a duplicate error when trying to
signal a non-existent job. This removes the second error.

No change in logic

Previous logic would not recognize a job ID specification with a
  job array task ID of "*" (e.g. "123_*") to indicated all job
  array tasks.
Previous logic would stop any parsing after the closing bracket
  on a job array specification (e.g. "123_[4-6] 234" would not
  see the "234").
Improve logging of job ID specifications (i.e. use job array
  specification).

Conflicts:
	NEWS

Modify slurmctld's parsing of a job_id string for the job_signal and
job_requeue calls to treat a job ID value of "#_*" as representing
all tasks in a job ID number "#". Previously treated as invalid input.

Also set the last_job_update time so that if a pending job is killed,
then that is reported immediately by "squeue -i#" (previously it
may keep reporting stale date.

Trying to avoid having technical questions sent to "sales@schedmd.com"