Do not test for CUDA environment variable on Cray system

needed for test24.1

Moe,

Here it is, I have added a subroutine to env.c to
unset the user's environment and then called it
from sbatch in main. I also removed the comment
from the sbatch man page indicating that it
wasn't working the same for a regular user as
it did for Moab. It should now be functionally
the same.

I think there is still a difference between how
sbatch functions with an environment in a file
than it does from when Moab excve'd the environment.
However, I'm not sure what it would be at this
point.

Again,  for so many iterations....

Phil

Moe,

After doing more extensive testing, I came to realize
that we had made a bad basic assumption. We believed
that the user's environment should only be what was
sent in the file via the --export-file option.

However, that broke the previous behavior, especially
in regard to Moab jobs. It also caused the SLURM
defined environment variables to be lost as well.

This patch will enable the correct behavior for Moab
on top of SLURM whne using the --export-file option,
but the behavior is less that pefect for using it
stand alone with sbatch. When using the option with
sbatch as a user, the file environment is read in,
and then when the env_array_merge is made, some
variables may get overwritten. This is good for
the SLURM and MPI vairables, but not so good for
others., The problem is trying to reconcile two
sources of environment is very problematic.

I also added a caveat in the man page.

I made changes in my branch of SchedMD SLURM
for 2.3, here is the patch.

Phil

Fix for possible invalid memory reference in slurmctld in job dependency
logic. Patch from Carles Fenoy (Barcelona Supercomputer Center).

all jobs would be returned even if the flag was set.
Patch from Bill Brophy, Bull.

We replaced references to "pipe" with a more generic "file descriptor".
We also replaced a while loop in env.c with a for loop.

Fix bug in --switch option with topology resulting in bad switch count use.
Patch from Alejandro Lucero Palau (Barcelona Supercomputer Center).

Add test7.15 to test for any signals blocked by default in job tasks.

It was found that slurmstepd was intermittently leaving SIGPIPE
blocked when launching user tasks. This may have something to do
with the fact that the xsignal_unblock() call in _fork_all_tasks()
is referencing an extern array (nominally this should have unblocked
SIGPIPE), but I didn't spend the time to fully track this issue
down. Instead, I figured there is probably no reason we would _not_
want to unblock *all* signals, so this patch does that.

Before this change, the following program fails every once in awhile:

 #include <stdio.h>
 #include <signal.h>

 int main (int ac, char **av)
 {
	int i, rc = 0;
	struct sigaction act;
	for (i = 1; i < SIGRTMAX; i++) {
		sigaction (i, NULL, &act);
		if (act.sa_handler == SIG_DFL)
			continue;
		fprintf (stderr, "Signal %d appears to be ignored!\n", i);
		rc = 1;
	}
	return (rc);
 }

with:

 srun -N1 -n1 ./test
 Signal 13 appears to be ignored!

after the change, the program succeeds.

number.

This new sbatch option allows the user to specify a file containing
environment variables to be used in the jobs execution environment.

Let operators see reservation data even if "PrivateData=reservations" flag
is set in slurm.conf. Patch from Don Albert, Bull.

It used to be listed as "--switch=" rather than "--switches="

Fix bug in srun --multi-prog configuration file to avoid printing duplicate
record error when "*" is used at the end of the file for the task ID. It
means all task IDs not otherwise identified.

Fix race condition where sbcast command can result in deadlock of slurmd
daemon. Patch by Don Albert, Bull.

In commit

 commit b33cd7c8
 Author: Mark A. Grondona <mgrondona@llnl.gov>
 Date:   Fri Oct 7 13:52:43 2011 -0700

a minor bug was introduced because the io_dup_stdio() call was moved
out from under the 'else' case of the empty if

 if (job->pty && (task->gtid == 0))

in task.c:exec_task() This results in the error

 dup2(stdin): Bad file descriptor

For task 0 when --pty is used, because io_dup_stdio() is
inappropriately being called after login_tty().

This patch collects the stdio setup into prepare_tty(), renames
that function prepare_stdio(), and removes the io_dup_stdio()
call from exec_task().  This removes the last io_* call from
slurmstepd/task.c, so the io.h header is removed.

I didn't see a problem with moving the io_dup_stdio() call earlier in
the task setup path, but it could probably use another set of eyes.
----------
Note: This problem was at least partly fixed with
https://github.com/SchedMD/slurm/commit/4d17300c5de7a8f146072bae8baaf773ca1fe98a

When importing an exported shell function, the bracket count
code didn't handle negative return value from _bracket_cnt(),
so a shell function (or a shell variable starting with '(')
would loop until the end of the env output.

Since the test for no open bracket wasn't handled anyway, remove
this and clean up _bracket_cnt() code (which now returns negative
if there are more closing than opening brackets).

Also, remove the unneded local integer bracket_cnt from the while
loops and only continue processing while the bracket tally is > 0.

Note: There are still many unhandled corner cases in this code.

src/common/env.c:env_array_user_default() tries to import a "default"
user environment by parsing the output of something like

 su - <user> -c "/bin/env" ( or "slurmstepd getenv")

(or an equivalent) when the job is run with --get-user-env. There is some
code that tries to detect and import exported bash shell functions, which
usually have the form

 func=() { ...
 }

into the job environment. However, one of the many problems with the
code is that the test for bounds overflow:

              if ((strlen(value) + strlen(line)) >
                  (sizeof(value) - 1))
                     break;

useis sizeof(value) and value is a char *. This means that all multiline
shell functions are truncated, and the following error appears in
users batch script output:

 /bin/sh: func: line 1: syntax error: unexpected end of file
 /bin/sh: error importing function definition for `func'

This patch fixes the bounds test.

Similar cut-and-paste code in _load_env_cache() was also fixed.

However, it should be noted that there are many other potential cases
where shell functions will not be properly imported by this code.
(For example, quoted unbalanced brackets used in the function)
Someday this should be fixed.

Prevent resetting a held job's priority when updating other job parameters.
Patch from Alejandro Lucero Palau, BSC.

Patch from John Thiltges, University of Nebraska-Lincoln.