JuliaDataCubes / JuliaDataCubes/YAXArrays.jl
DataFrame generation of CubeTable fails when dimensions are not aligned
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 132
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
When I try to convert a CubeTable object whose Lon and Lat axis is on position 3 and 4 to a Data Frame this fails with the following error:
julia> cttiny = CubeTable(;palfreqtiny, palnotiny, metrics=cubetiny)
Datacube iterator with 432 elements with fields: ("palfreqtiny", "palnotiny", "metrics")
julia> dftiny = DataFrame(cttiny)
ERROR: BoundsError: attempt to access 3×3×8×6 view(::Zarr.ZArray{Union{Missing, Float32}, 4, Zarr.BloscCompressor, Zarr.DirectoryStore}, Base.OneTo(3), Base.OneTo(3), 4103:4110, 3112:3117) with eltype Union{Missing, Float32} at index [1:8, 1:6, 1:3, 1:3]
Stacktrace:
[1] throw_boundserror(A::SubArray{Union{Missing, Float32}, 4, Zarr.ZArray{Union{Missing, Float32}, 4, Zarr.BloscCompressor, Zarr.DirectoryStore}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, UnitRange{Int64}, UnitRange{Int64}}, false}, I::NTuple{4, UnitRange{Int64}})
@ Base ./abstractarray.jl:651
[2] checkbounds
@ ./abstractarray.jl:616 [inlined]
[3] view(::SubArray{Union{Missing, Float32}, 4, Zarr.ZArray{Union{Missing, Float32}, 4, Zarr.BloscCompressor, Zarr.DirectoryStore}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, UnitRange{Int64}, UnitRange{Int64}}, false}, ::UnitRange{Int64}, ::UnitRange{Int64}, ::UnitRange{Int64}, ::UnitRange{Int64})
@ Base ./subarray.jl:177
[4] readblock!(::DiskArrays.SubDiskArray{Union{Missing, Float32}, 4}, ::Array{Union{Missing, Float32}, 4}, ::UnitRange{Int64}, ::Vararg{UnitRange{Int64}, N} where N)
@ DiskArrays ~/.julia/dev/DiskArrays/src/subarrays.jl:14
[5] getindex_disk(::DiskArrays.SubDiskArray{Union{Missing, Float32}, 4}, ::UnitRange{Int64}, ::Vararg{UnitRange{Int64}, N} where N)
@ DiskArrays ~/.julia/dev/DiskArrays/src/DiskArrays.jl:60
[6] getindex(::DiskArrays.SubDiskArray{Union{Missing, Float32}, 4}, ::UnitRange{Int64}, ::UnitRange{Int64}, ::UnitRange{Int64}, ::UnitRange{Int64})
@ DiskArrays ~/.julia/dev/DiskArrays/src/DiskArrays.jl:182
[7] updatear(f::Symbol, r::NTuple{4, UnitRange{Int64}}, cube::YAXArray{Union{Missing, Float32}, 4, DiskArrays.SubDiskArray{Union{Missing, Float32}, 4}, Vector{CubeAxis}}, indscol::Vector{Int64}, loopinds::Vector{Int64}, cache::Array{Union{Missing, Float32}, 4})
@ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:399
[8] (::YAXArrays.DAT.var"#72#73"{NTuple{4, UnitRange{Int64}}, Symbol})(ic::YAXArrays.DAT.InputCube{4}, ca::Array{Union{Missing, Float32}, 4})
@ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:348
[9] foreach(::Function, ::Tuple{YAXArrays.DAT.InputCube{2}, YAXArrays.DAT.InputCube{2}, YAXArrays.DAT.InputCube{4}}, ::Tuple{Matrix{Float32}, Matrix{Float32}, Array{Union{Missing, Float32}, 4}})
@ Base ./abstractarray.jl:2142
[10] updatears(clist::Tuple{YAXArrays.DAT.InputCube{2}, YAXArrays.DAT.InputCube{2}, YAXArrays.DAT.InputCube{4}}, r::NTuple{4, UnitRange{Int64}}, f::Symbol, caches::Tuple{Matrix{Float32}, Matrix{Float32}, Array{Union{Missing, Float32}, 4}})
@ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:346
[11] iterate(ci::YAXArrays.DAT.CubeIterator{Array{NTuple{4, UnitRange{Int64}}, 4}, Tuple{Matrix{Float32}, Matrix{Float32}, Array{Union{Missing, Float32}, 4}}, Tuple{YAXArrays.YAXTools.PickAxisArray{Float32, 2, Matrix{Float32}, (1, 2), nothing}, YAXArrays.YAXTools.PickAxisArray{Float32, 2, Matrix{Float32}, (1, 2), nothing}, YAXArrays.YAXTools.PickAxisArray{Union{Missing, Float32}, 4, Array{Union{Missing, Float32}, 4}, (1, 2, 3, 4), nothing}}, Tuple{RangeAxis{Float64, :Lon, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, RangeAxis{Float64, :Lat, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{11, UInt8}, :Distfuns, Vector{Zarr.MaxLengthStrings.MaxLengthString{11, UInt8}}}, CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{6, UInt8}, :FrequencyBins, Vector{Zarr.MaxLengthStrings.MaxLengthString{6, UInt8}}}}, (), NamedTuple{(:palfreqtiny, :palnotiny, :metrics), Tuple{YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}, YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}, YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}}}})
@ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/dciterators.jl:70
[12] iterate
@ ./iterators.jl:159 [inlined]
[13] iterate
@ ./iterators.jl:158 [inlined]
[14] buildcolumns
@ ~/.julia/packages/Tables/uYJXY/src/fallbacks.jl:134 [inlined]
[15] columns
@ ~/.julia/packages/Tables/uYJXY/src/fallbacks.jl:253 [inlined]
[16] DataFrame(x::YAXArrays.DAT.CubeIterator{Array{NTuple{4, UnitRange{Int64}}, 4}, Tuple{Matrix{Float32}, Matrix{Float32}, Array{Union{Missing, Float32}, 4}}, Tuple{YAXArrays.YAXTools.PickAxisArray{Float32, 2, Matrix{Float32}, (1, 2), nothing}, YAXArrays.YAXTools.PickAxisArray{Float32, 2, Matrix{Float32}, (1, 2), nothing}, YAXArrays.YAXTools.PickAxisArray{Union{Missing, Float32}, 4, Array{Union{Missing, Float32}, 4}, (1, 2, 3, 4), nothing}}, Tuple{RangeAxis{Float64, :Lon, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, RangeAxis{Float64, :Lat, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{11, UInt8}, :Distfuns, Vector{Zarr.MaxLengthStrings.MaxLengthString{11, UInt8}}}, CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{6, UInt8}, :FrequencyBins, Vector{Zarr.MaxLengthStrings.MaxLengthString{6, UInt8}}}}, (), NamedTuple{(:palfreqtiny, :palnotiny, :metrics), Tuple{YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}, YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}, YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}}}}; copycols::Bool)
@ DataFrames ~/.julia/packages/DataFrames/3mEXm/src/other/tables.jl:58
[17] DataFrame(x::YAXArrays.DAT.CubeIterator{Array{NTuple{4, UnitRange{Int64}}, 4}, Tuple{Matrix{Float32}, Matrix{Float32}, Array{Union{Missing, Float32}, 4}}, Tuple{YAXArrays.YAXTools.PickAxisArray{Float32, 2, Matrix{Float32}, (1, 2), nothing}, YAXArrays.YAXTools.PickAxisArray{Float32, 2, Matrix{Float32}, (1, 2), nothing}, YAXArrays.YAXTools.PickAxisArray{Union{Missing, Float32}, 4, Array{Union{Missing, Float32}, 4}, (1, 2, 3, 4), nothing}}, Tuple{RangeAxis{Float64, :Lon, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, RangeAxis{Float64, :Lat, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{11, UInt8}, :Distfuns, Vector{Zarr.MaxLengthStrings.MaxLengthString{11, UInt8}}}, CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{6, UInt8}, :FrequencyBins, Vector{Zarr.MaxLengthStrings.MaxLengthString{6, UInt8}}}}, (), NamedTuple{(:palfreqtiny, :palnotiny, :metrics), Tuple{YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}, YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}, YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}}}})
@ DataFrames ~/.julia/packages/DataFrames/3mEXm/src/other/tables.jl:49
[18] top-level scope
@ REPL[285]:1
The size of the corresponding cubes have the following axes and sizes:
julia> palfreqtiny
YAXArray with the following dimensions
Lon Axis with 8 Elements from 684669.15 to 684879.15
Lat Axis with 6 Elements from 9.39391066e6 to 9.39376066e6
Total size: 192.0 bytes
julia> palnotiny
YAXArray with the following dimensions
Lon Axis with 8 Elements from 684669.15 to 684879.15
Lat Axis with 6 Elements from 9.39391066e6 to 9.39376066e6
Total size: 192.0 bytes
julia> cubetiny
YAXArray with the following dimensions
Distfuns Axis with 3 elements: euclidean crosscormax cityblock
FrequencyBins Axis with 3 elements: Fast Annual Slow
Lon Axis with 8 Elements from 684669.15 to 684879.15
Lat Axis with 6 Elements from 9.39391066e6 to 9.39376066e6
Total size: 1.69 KB
If I permute cubetiny so that the Lon Axis is the first and Lat is the second axis, the dataframe generation works.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure at the DataFrame(cttiny) entry point using a CubeTable whose Lon and Lat axes are in positions 3 and 4. Read the YAXArrays.DAT iterator path shown in the stack trace, especially updatear and dciterators.jl, and compare it with the working permuted case. Done means DataFrame conversion works without requiring Lon and Lat to be the first two axes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100