Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • autosubmit autosubmit
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 338
    • Issues 338
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 21
    • Merge requests 21
  • Deployments
    • Deployments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • Earth SciencesEarth Sciences
  • autosubmitautosubmit
  • Wiki
  • migrate experiments

migrate experiments · Changes

Page history
Update migrate experiments authored Jun 03, 2021 by dbeltran's avatar dbeltran
Show whitespace changes
Inline Side-by-side
migrate-experiments.md
View page @ f1226a6b
...@@ -22,11 +22,67 @@ Before starting the migation, the following actions are recommended: ...@@ -22,11 +22,67 @@ Before starting the migation, the following actions are recommended:
# Configure your experiment for the migration # Configure your experiment for the migration
(TODO: remind to change platforms/queues) In order to migrate an experiment, an configuration of the migrate parameters for EACH platform that the experiment uses is necessary. You must set all migrate parameters on platforms_expid.conf.
The mandatory configuration consists of:
* USER_TO = <target_user>
* TEMP_DIR = <hpc_temporary_directory> *
Temp_dir is a dir on the remote platform to which both users must have access. Ideally, this dir should have these permissions (RWX|RWX|---), however (RWX|RWX|RWX) also works.
Another important point to have into account is that if your experiment data is in a shared filesystem ( example, dt-transfer and marenostrum4) you only have to fill TEMP_DIR on the dt-transfer platform.
ex:
```
[marenostrum4]
# Queue type. Options: ps, SGE, LSF, SLURM, PBS, eceaccess
TYPE = slurm
HOST = mn3.bsc.es
PROJECT = bsc32
USER = bsc32070
QUEUE = debug
SCRATCH_DIR = /gpfs/scratch
ADD_PROJECT_TO_HOST = false
MAX_WALLCLOCK = 2:00
USER_TO = bsc32523
TEMP_DIR =
[marenostrum_archive]
TYPE = ps
HOST = dt02.bsc.es
PROJECT = bsc32
USER = bsc32070
SCRATCH_DIR = /gpfs/scratch
ADD_PROJECT_TO_HOST = False
TEST_SUITE = False
USER_TO = bsc32523
TEMP_DIR = /gpfs/scratch/bsc32/bsc32070/temp_dir
```
Finally, is mandatory that this folder is in the same filesystem.
You can expand this basic configuration as follow:
* SAME_USER = false|true # Default, FALSE, used for mantain the same remote_user
* PROJECT_TO = <project> # Optional, if not specified project will remain the same
* HOST_TO = <cluster_ip> # Optional, avoid alias if possible, try use direct ip.
Before running the experiment with the new user, remind them to change queue parameter (example CLASS_A to bsc_es) if necessary.
# Perform the migration # Perform the migration
(TODO: remind to use 3.13.0) Once configured, double-check that you're using the AS v3.13.0+.
There are two modes of running `autosubmit migrate`, both of them consist of a two-step procedure:
* Migrate everything:
* (User-Owner) autosubmit migrate <expid> -o
* (User-Target)autosubmit migrate <expid> -p
* Migrate only remote files. (usefull if the local user is the same)
* (User-Owner) autosubmit migrate <expid> -o --onlyremote
* (User-Target) autosubmit migrate <expid> -p --onlyremote
# Tests done # Tests done
......
Clone repository
  • Code coverage
  • Deployment
  • Issues documenting different aspects
  • Leaflet
  • Possible Operational Problems and Solutions
  • Running Autosubmit in Earth Sciences
  • Testing_Suite
  • Updating ReadTheDocs Autosubmit documentation
  • Visual Identity
  • [DestinE] Autosubmit VM on Lumi
  • background
  • bibtex
  • databases
  • development
  • dissemination
View All Pages