Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • startR startR
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 29
    • Issues 29
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 7
    • Merge requests 7
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Earth SciencesEarth Sciences
  • startRstartR
  • Issues
  • #205
Closed
Open
Issue created Oct 28, 2024 by vagudets@vagudetsMaintainer

Use Compute() with conda environments on HPCs

Hi me,

Compute() currently has the arguments r_module and cdo_module that are used to specify the name of the R and CDO modules to be loaded on the HPC platform when launching jobs to the cluster. These arguments are passed to ByChunks_ecflow() or ByChunks_autosubmit() so that the modules are sourced when each chunk is run on the cluster.

However, in some machines we use conda environments rather than modules, and startR should still be able to work as long as R, the required R libraries and CDO are installed in the conda environment.

This would require adding a new argument for Compute() that could be called conda_environment with a NULL default value. The user can specify the conda environment and it can be loaded instead of the R and CDO modules. In order to do this:

  • The initial checks in ByChunks_*() need to be modified to include this new parameter and discard r_module and cdo_module if they won't be used.
  • A condition to activate the conda environment needs to be added to ByChunks_ecflow().
  • A condition to activate the conda environment needs to be added to write_autosubmit_bash() in Utils.R

IMPORTANT:

To properly load conda environments from inside bash scripts that are launched with slurm, the following lines are necessary:

set +eu
source /gpfs/projects/bsc32/software/suselinux/11/software/Miniconda3/4.7.10/etc/profile.d/conda.sh 
conda activate /gpfs/projects/bsc32/repository/apps/conda_envs/SUNSET-env_2.0.0
set -eu

The second line (source .../conda.sh) is platform-dependent; ideally we should find a universal solution.

Victòria

Edited Mar 26, 2025 by vagudets
Assignee
Assign to
Time tracking