dat dimension cannot be chunked
When two datasets are loaded together and the chunking dimension includes "dat", the error is returned:
Error in get_chunk_indices(length(dat_selectors[[file_dim]][[j]]), chunks[[file_dim]]["chunk"], : Requested to divide dimension 'dat' of length 1 in 2 chunks, which is not possible.
It is not unexpected because startR loads datasets one after another (in a for loop), and it is not necessary because we can use two Start() calls to define two datasets separately, if they are independent (therefore, can be chunked). But it should return a more meaningful error.
The following script can produce this error.
library(startR)
path1 <- "/esarchive/exp/ecmwf/system5c3s/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
path2 <- "/esarchive/exp/ecmwf/system5_m1/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
data <- Start(dat = list(list(name = 'system5c3s', path = path1),
list(name = 'system5_m1', path = path2)),
var = c('tas'),
sdate = paste0(2017:2018, '0501'),
ensemble = 'all',
time = indices(1:3),
lat = values(list(20, 80)), lat_reorder = Sort(),
lon = values(list(-80, 40)), lon_reorder = CircularSort(-180, 180),
transform = CDORemapper,
transform_extra_cells = 2,
transform_params = list(grid = 'r360x181',
method = 'conservative'),
transform_vars = c('lat', 'lon'),
synonims = list(lat = c('lat', 'latitude'), lon = c('lon', 'longitude')),
return_vars = list(time = 'sdate', lon = 'dat', lat = 'dat'),
retrieve = FALSE)
func <- function(x) {
return(x)
}
step <- Step(func, target_dims = c('lat', 'lon'), output_dims = c('lat', 'lon'))
wf <- AddStep(data, step)
res <- Compute(wf, chunks = list(sdate = 2, dat = 2))