Folder in scratch does not follow the pattern %SCRATCH_DIR%/%CURRENT_PROJ% in Juwels

Reported in: mattermost/MPI-M/Autosubmit-support

Tag: v4.0.0b

Hi all,

In Juwels, the folders associated to the projects (accounts) do not match the project name. For example, the account I use in Juwels is hhb19 for #SBATCH -A, however, the project's scratch is /p/scratch/chhb (an extra c to the project's name).

This creates the following problem when one sets PROJECT: chhb19 in the platform file:

ERROR] Trace: sbatch: error: batch job submission failed: invalid account or account/partition combination specified

And the following problem if one sets it to hhb19 (because it's unable to create this file /p/scratch/hhb19/andres-martinez1/permission_checker_azxbyc when checking the connection due to the missing c):

Scripts OK
Autosubmit is running with v4.0.0
Checking the connection to all platforms in use
[local] Correct user privileges for host localhost
[local] Connection successful to host localhost
Traceback (most recent call last):
  File "/home/ollie/mandresm/utilities/autosubmit/autosubmit/autosubmit.py", line 1858, in run_experiment
    Autosubmit.restore_platforms(platforms_to_test)
  File "/home/ollie/mandresm/utilities/autosubmit/autosubmit/autosubmit.py", line 2253, in restore_platforms
    "Issues while checking the connectivity of platforms.", 7010, issues + "\n" + ssh_config_issues)
log.log.AutosubmitCritical:  

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ollie/mandresm/utilities/autosubmit/bin/autosubmit", line 36, in main
    Autosubmit.parse_args()
  File "/home/ollie/mandresm/utilities/autosubmit/autosubmit/autosubmit.py", line 630, in parse_args
    args.start_after, args.run_members)
  File "/home/ollie/mandresm/utilities/autosubmit/autosubmit/autosubmit.py", line 2202, in run_experiment
    raise AutosubmitCritical(e.message, e.code, e.trace)
log.log.AutosubmitCritical:  
[ERROR] None
[ERROR] Trace: 
[JUWELS] has configuration issues.
 Check that the connection is passwd-less.(ssh andres-martinez1@juwels-cluster)
 Check the parameters that build the root_path are correct:{scratch_dir/project/user} = {/p/scratch//hhb19/andres-martinez1}

 [CRITICAL] Issues while checking the connectivity of platforms. [eCode=7010]

Is all this a real problem, or is there an option that allows to overcome it already?

I have overcome the problem and managed to get the full vanilla workflow working by adding a new variable in the platform file named SCRATCH_PROJECT_DIR: chhb19 while PROJECT: hhb19. I also created a new property in the platform object: platform.project_dir which takes values of SCRATCH_PROJECT_DIR if it exists and of PROJECT if it doesn't. Then I use this variable everywhere where platform.project was used for the project directory in scratch.

I have tried to push my branch to do a PR in case it's useful for the more generalized solution you might want to produced, but I don't think I have the permissions.