Skip to content
GitLab

Collect() result from cluster

Closed Nuria Pérez-Zanón requested to merge develop-Collect into master

Hi @aho,

This merge request contains a first version to Collect() results from cluster. It would be good to discuss about the warnings and cleaning it should contain. For now, I have run a use case (https://earth.bsc.es/gitlab/es/startR/-/blob/master/inst/doc/usecase/ex2_8_calibration.R) and tested it worked:

Work station:

res <- Compute(wf,
                     chunks = list(latitude = 2, longitude = 2),
                     threads_load = 2,
                     threads_compute = 4,
                     cluster = list(queue_host = "nord3",
                                    queue_type = 'slurm',
                                    cores_per_job = 2,
                                    temp_dir = '/gpfs/scratch/bsc32/bsc32339/startR_hpc/',
                                    polling_period = 10,
                                    job_wallclock = '01:00:00',
                                    max_jobs = 4,
                                    bidirectional = FALSE),
                     ecflow_suite_dir = "/esarchive/scratch/nperez/ecflow", 
              wait = FALSE) # your path!

saveRDS(res, file = '/esarchive/scratch/nperez/test_collect.Rds')

Nord3:

res <- readRDS('/esarchive/scratch/nperez/test_collect.Rds')
source("/esarchive/scratch/nperez/git/startR/R/Collect.R")
result <- Collect(res, wait = TRUE, remove = F)

Cheers,

Núria