JuliaDataCubes / JuliaDataCubes/YAXArrays.jl
Cache miss leads for distributed savecube leads to KeyError
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 132
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
When I try to save a too large zarr file into a different chunking, I run into the following error:
This only happens, when I try to do that in parallel.
julia> savecube(vvcube, "data/cubes/s1cube_jurua_vv_smallchunks.zarr", chunksize=Dict("X"=>30, "Y"=>30, "Time"=>100), overwrite=true, max_cache=10^8)
ispar = true
ispar = true
┌ Warning: There are still cache misses
└ @ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:1004
nopar = true
dc.ispar = false
ERROR: KeyError: key 1 not found
Stacktrace:
[1] getindex(h::Dict{Int64, Int64}, key::Int64)
@ Base ./dict.jl:482
[2] (::YAXArrays.DAT.var"#147#149"{YAXArrays.DAT.DATConfig{1, 1}})(i::YAXArrays.DAT.InputCube{3})
@ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:1022
[3] map
@ ./tuple.jl:213 [inlined]
[4] generateworkarrays(dc::YAXArrays.DAT.DATConfig{1, 1})
@ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:1024
[5] getallargs(dc::YAXArrays.DAT.DATConfig{1, 1})
@ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:677
[6] runLoop(dc::YAXArrays.DAT.DATConfig{1, 1}, showprog::Bool)
@ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:648
[7] mapCube(::YAXArrays.DAT.var"#cop#269", ::Tuple{YAXArray{Union{Missing, Float32}, 3, ZArray{Union{Missing, Float32}, 3, Zarr.BloscCompressor, DirectoryStore}, Vector{RangeAxis}}}; max_cache::Int64, indims::InDims, outdims::OutDims, inplace::Bool, ispar::Bool, debug::Bool, include_loopvars::Bool, showprog::Bool, nthreads::Dict{Int64, Int64}, loopchunksize::Dict{Any, Any}, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:466
[8] #mapCube#38
@ ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:311 [inlined]
[9] savecube(c::YAXArray{Union{Missing, Float32}, 3, ZArray{Union{Missing, Float32}, 3, Zarr.BloscCompressor, DirectoryStore}, Vector{RangeAxis}}, name::String; chunksize::Dict{String, Int64}, max_cache::Int64, backend::Symbol, backendargs::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:overwrite,), Tuple{Bool}}})
@ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/CubeIO.jl:63
[10] top-level scope
@ REPL[25]:1
I think, the problem is, that the chunk miss sets the ispar variable of the `DATConfig` struct to false, while the nthreads information is still for the parallel case. Maybe it would be enough to set the `dc.nthreads` variable after the cache miss check.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the parallel savecube call from the issue, then inspect the cache-miss handling around DAT/DAT.jl:1004 and generateworkarrays at DAT.jl:1022. Trace how ispar and nthreads are used after the cache-miss check, with CubeIO.jl:63 as the savecube entry point. Done means the reported parallel savecube operation completes without the KeyError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100