Hi @aho!
This brach partially solves the problem.
I have installed this branch and use it with your minimal example and the following Compute call:
res_n4 <- Compute(wf,
chunks = list(sdate = 1),
threads_load = 1,
threads_compute = 4,
cluster = list(queue_host = 'nord4',
queue_type = 'slurm',
temp_dir = temp_dir,
lib_dir =
"/.statelite/tmpfs/gpfs/home/bsc32/bsc32339/R/x86_64-pc-linux-gnu-library/3.6",
cores_per_job = 2,
job_wallclock = '01:00:00',
max_jobs = 4,
bidirectional = FALSE,
polling_period = 10
),
ecflow_suite_dir = ecflow_suite_dir,
wait = T)
The result is correct in this case:
summary(res_n4$output1)
Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
244.6 251.4 256.9 259.9 272.3 274.0 8
However, when setting threads_load = 2
it fails:
* Loading... This may take several minutes...
* Progress: 0%Error in parallel::clusterApplyLB(cluster, work_pieces, .LoadDataFile, :
object 'transform_crop_domain' not found
In addition: Warning messages:
1: ! Warning: Parameter 'pattern_dims' not specified. Taking the first dimension,
! 'dat' as 'pattern_dims'.
2: ! Warning: Could not find any pattern dim with explicit data set descriptions (in
! the form of list of lists). Taking the first pattern dim, 'dat', as
! dimension with pattern specifications.
3: In filebacked.big.matrix(nrow = nrow, ncol = ncol, type = type, :
No descriptor file given, it will be named _2842325790_1_1_1_1_1_1_.desc
The data cannot be loaded.
See the original error message:
object 'found_files' not found
Well, I'll keep testing it.
Cheers,
Núria