JuliaDataCubes / JuliaDataCubes/YAXArrays.jl

Error during distributed mapslices on GDAL (vrt) cube

Open
#80 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
132
Forks
25
PR merge metrics
No merged PRs in 30d

Description

Hi all,
I've tried to apply a simple function (mean over time) to a GDAL based cube and got the following error:

stat = mapslices(cube_mean, cube, dims="Time")

 From worker 3:    Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")
      From worker 5:    Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")
      From worker 2:    Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")
      From worker 4:    Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")ERROR: 
GDALError (CE_Failure, code 10):
        Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.


Stacktrace:
 [1] runLoop(dc::YAXArrays.DAT.DATConfig{1, 1}, showprog::Bool)
   @ YAXArrays.DAT ~/.julia/packages/YAXArrays/HzZlX/src/DAT/DAT.jl:638
 [2] mapCube(::typeof(cube_mean), ::Tuple{YAXArray{UInt16, 3, ArchGDAL.RasterDataset{UInt16, ArchGDAL.IDataset}, Vector{CubeAxis}}}; max_cache::Float64, indims::InDims, outdims::OutDims, inplace::Bool, ispar::Bool, debug::Bool, include_loopvars::Bool, showprog::Bool, nthreads::Dict{Int64, Int64}, loopchunksize::Dict{Any, Any}, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ YAXArrays.DAT ~/.julia/packages/YAXArrays/HzZlX/src/DAT/DAT.jl:423
 [3] #mapCube#40
   @ ~/.julia/packages/YAXArrays/HzZlX/src/DAT/DAT.jl:270 [inlined]
 [4] #mapslices#54
   @ ~/.julia/packages/YAXArrays/HzZlX/src/DAT/DAT.jl:329 [inlined]
 [5] top-level scope
   @ REPL[58]:1

I tried @eval @everywhere cube = $cube but without success.
Any ideas if that can be fixed easily?

Thanks for your work and best regards!
Björn

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the runLoop and mapCube entries in YAXArrays.DAT/DAT.jl, especially the stack-trace locations around lines 638 and 423, and reproduce mapslices(cube_mean, cube, dims="Time") on the GDAL VRT cube. Investigate why distributed workers reach GDALDatasetRasterIOEx with a NULL hDS; done means the distributed mapslices call completes without that error.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.