Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • startR startR
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 29
    • Issues 29
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 7
    • Merge requests 7
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Earth SciencesEarth Sciences
  • startRstartR
  • Issues
  • #180
Closed
Open
Issue created Jul 25, 2023 by aho@ahoMaintainer

dat dimension cannot be chunked

When two datasets are loaded together and the chunking dimension includes "dat", the error is returned:

Error in get_chunk_indices(length(dat_selectors[[file_dim]][[j]]), chunks[[file_dim]]["chunk"], : Requested to divide dimension 'dat' of length 1 in 2 chunks, which is not possible.

It is not unexpected because startR loads datasets one after another (in a for loop), and it is not necessary because we can use two Start() calls to define two datasets separately, if they are independent (therefore, can be chunked). But it should return a more meaningful error.

The following script can produce this error.

  library(startR)
  path1 <- "/esarchive/exp/ecmwf/system5c3s/monthly_mean/$var$_f6h/$var$_$sdate$.nc"
  path2 <- "/esarchive/exp/ecmwf/system5_m1/monthly_mean/$var$_f6h/$var$_$sdate$.nc"

  data <- Start(dat = list(list(name = 'system5c3s', path = path1),
                           list(name = 'system5_m1', path = path2)),
                var = c('tas'),
                sdate = paste0(2017:2018, '0501'),
                ensemble = 'all',
                time = indices(1:3),
                lat = values(list(20, 80)), lat_reorder = Sort(),
                lon = values(list(-80, 40)), lon_reorder = CircularSort(-180, 180),
                transform = CDORemapper,
                transform_extra_cells = 2,
                transform_params = list(grid = 'r360x181',
                                        method = 'conservative'),
                transform_vars = c('lat', 'lon'),
                synonims = list(lat = c('lat', 'latitude'), lon = c('lon', 'longitude')),
                return_vars = list(time = 'sdate', lon = 'dat', lat = 'dat'),
                retrieve = FALSE)
 func <- function(x) {
   return(x)
 }
 step <- Step(func, target_dims = c('lat', 'lon'), output_dims = c('lat', 'lon'))
 wf <- AddStep(data, step)
 res <- Compute(wf, chunks = list(sdate = 2, dat = 2))
Assignee
Assign to
Time tracking