Folder in scratch does not follow the pattern %SCRATCH_DIR%/%CURRENT_PROJ% in Juwels
Reported in: mattermost/MPI-M/Autosubmit-support
Tag: v4.0.0b
Hi all,
In Juwels, the folders associated to the projects (accounts) do not match the project name. For example, the account I use in Juwels is hhb19
for #SBATCH -A
, however, the project's scratch is /p/scratch/chhb
(an extra c to the project's name).
This creates the following problem when one sets PROJECT: chhb19
in the platform file:
ERROR] Trace: sbatch: error: batch job submission failed: invalid account or account/partition combination specified
And the following problem if one sets it to hhb19
(because it's unable to create this file /p/scratch/hhb19/andres-martinez1/permission_checker_azxbyc
when checking the connection due to the missing c):
Scripts OK
Autosubmit is running with v4.0.0
Checking the connection to all platforms in use
[local] Correct user privileges for host localhost
[local] Connection successful to host localhost
Traceback (most recent call last):
File "/home/ollie/mandresm/utilities/autosubmit/autosubmit/autosubmit.py", line 1858, in run_experiment
Autosubmit.restore_platforms(platforms_to_test)
File "/home/ollie/mandresm/utilities/autosubmit/autosubmit/autosubmit.py", line 2253, in restore_platforms
"Issues while checking the connectivity of platforms.", 7010, issues + "\n" + ssh_config_issues)
log.log.AutosubmitCritical:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ollie/mandresm/utilities/autosubmit/bin/autosubmit", line 36, in main
Autosubmit.parse_args()
File "/home/ollie/mandresm/utilities/autosubmit/autosubmit/autosubmit.py", line 630, in parse_args
args.start_after, args.run_members)
File "/home/ollie/mandresm/utilities/autosubmit/autosubmit/autosubmit.py", line 2202, in run_experiment
raise AutosubmitCritical(e.message, e.code, e.trace)
log.log.AutosubmitCritical:
[ERROR] None
[ERROR] Trace:
[JUWELS] has configuration issues.
Check that the connection is passwd-less.(ssh andres-martinez1@juwels-cluster)
Check the parameters that build the root_path are correct:{scratch_dir/project/user} = {/p/scratch//hhb19/andres-martinez1}
[CRITICAL] Issues while checking the connectivity of platforms. [eCode=7010]
Is all this a real problem, or is there an option that allows to overcome it already?
I have overcome the problem and managed to get the full vanilla workflow working by adding a new variable in the platform file named SCRATCH_PROJECT_DIR: chhb19
while PROJECT: hhb19
. I also created a new property in the platform object: platform.project_dir
which takes values of SCRATCH_PROJECT_DIR
if it exists and of PROJECT
if it doesn't. Then I use this variable everywhere where platform.project
was used for the project directory in scratch.
I have tried to push my branch to do a PR in case it's useful for the more generalized solution you might want to produced, but I don't think I have the permissions.