JuliaDataCubes / JuliaDataCubes/YAXArrays.jl

DataFrame generation of CubeTable fails when dimensions are not aligned

Open
#53 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

test needed
Dominant language
Julia
Stars
132
Forks
25
PR merge metrics
No merged PRs in 30d

Description

When I try to convert a CubeTable object whose Lon and Lat axis is on position 3 and 4 to a Data Frame this fails with the following error:

julia> cttiny = CubeTable(;palfreqtiny, palnotiny, metrics=cubetiny)
Datacube iterator with 432 elements with fields: ("palfreqtiny", "palnotiny", "metrics")

julia> dftiny = DataFrame(cttiny)
ERROR: BoundsError: attempt to access 3×3×8×6 view(::Zarr.ZArray{Union{Missing, Float32}, 4, Zarr.BloscCompressor, Zarr.DirectoryStore}, Base.OneTo(3), Base.OneTo(3), 4103:4110, 3112:3117) with eltype Union{Missing, Float32} at index [1:8, 1:6, 1:3, 1:3]
Stacktrace:
  [1] throw_boundserror(A::SubArray{Union{Missing, Float32}, 4, Zarr.ZArray{Union{Missing, Float32}, 4, Zarr.BloscCompressor, Zarr.DirectoryStore}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, UnitRange{Int64}, UnitRange{Int64}}, false}, I::NTuple{4, UnitRange{Int64}})
    @ Base ./abstractarray.jl:651
  [2] checkbounds
    @ ./abstractarray.jl:616 [inlined]
  [3] view(::SubArray{Union{Missing, Float32}, 4, Zarr.ZArray{Union{Missing, Float32}, 4, Zarr.BloscCompressor, Zarr.DirectoryStore}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, UnitRange{Int64}, UnitRange{Int64}}, false}, ::UnitRange{Int64}, ::UnitRange{Int64}, ::UnitRange{Int64}, ::UnitRange{Int64})
    @ Base ./subarray.jl:177
  [4] readblock!(::DiskArrays.SubDiskArray{Union{Missing, Float32}, 4}, ::Array{Union{Missing, Float32}, 4}, ::UnitRange{Int64}, ::Vararg{UnitRange{Int64}, N} where N)
    @ DiskArrays ~/.julia/dev/DiskArrays/src/subarrays.jl:14
  [5] getindex_disk(::DiskArrays.SubDiskArray{Union{Missing, Float32}, 4}, ::UnitRange{Int64}, ::Vararg{UnitRange{Int64}, N} where N)
    @ DiskArrays ~/.julia/dev/DiskArrays/src/DiskArrays.jl:60
  [6] getindex(::DiskArrays.SubDiskArray{Union{Missing, Float32}, 4}, ::UnitRange{Int64}, ::UnitRange{Int64}, ::UnitRange{Int64}, ::UnitRange{Int64})
    @ DiskArrays ~/.julia/dev/DiskArrays/src/DiskArrays.jl:182
  [7] updatear(f::Symbol, r::NTuple{4, UnitRange{Int64}}, cube::YAXArray{Union{Missing, Float32}, 4, DiskArrays.SubDiskArray{Union{Missing, Float32}, 4}, Vector{CubeAxis}}, indscol::Vector{Int64}, loopinds::Vector{Int64}, cache::Array{Union{Missing, Float32}, 4})
    @ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:399
  [8] (::YAXArrays.DAT.var"#72#73"{NTuple{4, UnitRange{Int64}}, Symbol})(ic::YAXArrays.DAT.InputCube{4}, ca::Array{Union{Missing, Float32}, 4})
    @ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:348
  [9] foreach(::Function, ::Tuple{YAXArrays.DAT.InputCube{2}, YAXArrays.DAT.InputCube{2}, YAXArrays.DAT.InputCube{4}}, ::Tuple{Matrix{Float32}, Matrix{Float32}, Array{Union{Missing, Float32}, 4}})
    @ Base ./abstractarray.jl:2142
 [10] updatears(clist::Tuple{YAXArrays.DAT.InputCube{2}, YAXArrays.DAT.InputCube{2}, YAXArrays.DAT.InputCube{4}}, r::NTuple{4, UnitRange{Int64}}, f::Symbol, caches::Tuple{Matrix{Float32}, Matrix{Float32}, Array{Union{Missing, Float32}, 4}})
    @ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/DAT.jl:346
 [11] iterate(ci::YAXArrays.DAT.CubeIterator{Array{NTuple{4, UnitRange{Int64}}, 4}, Tuple{Matrix{Float32}, Matrix{Float32}, Array{Union{Missing, Float32}, 4}}, Tuple{YAXArrays.YAXTools.PickAxisArray{Float32, 2, Matrix{Float32}, (1, 2), nothing}, YAXArrays.YAXTools.PickAxisArray{Float32, 2, Matrix{Float32}, (1, 2), nothing}, YAXArrays.YAXTools.PickAxisArray{Union{Missing, Float32}, 4, Array{Union{Missing, Float32}, 4}, (1, 2, 3, 4), nothing}}, Tuple{RangeAxis{Float64, :Lon, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, RangeAxis{Float64, :Lat, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{11, UInt8}, :Distfuns, Vector{Zarr.MaxLengthStrings.MaxLengthString{11, UInt8}}}, CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{6, UInt8}, :FrequencyBins, Vector{Zarr.MaxLengthStrings.MaxLengthString{6, UInt8}}}}, (), NamedTuple{(:palfreqtiny, :palnotiny, :metrics), Tuple{YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}, YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}, YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}}}})
    @ YAXArrays.DAT ~/.julia/dev/YAXArrays/src/DAT/dciterators.jl:70
 [12] iterate
    @ ./iterators.jl:159 [inlined]
 [13] iterate
    @ ./iterators.jl:158 [inlined]
 [14] buildcolumns
    @ ~/.julia/packages/Tables/uYJXY/src/fallbacks.jl:134 [inlined]
 [15] columns
    @ ~/.julia/packages/Tables/uYJXY/src/fallbacks.jl:253 [inlined]
 [16] DataFrame(x::YAXArrays.DAT.CubeIterator{Array{NTuple{4, UnitRange{Int64}}, 4}, Tuple{Matrix{Float32}, Matrix{Float32}, Array{Union{Missing, Float32}, 4}}, Tuple{YAXArrays.YAXTools.PickAxisArray{Float32, 2, Matrix{Float32}, (1, 2), nothing}, YAXArrays.YAXTools.PickAxisArray{Float32, 2, Matrix{Float32}, (1, 2), nothing}, YAXArrays.YAXTools.PickAxisArray{Union{Missing, Float32}, 4, Array{Union{Missing, Float32}, 4}, (1, 2, 3, 4), nothing}}, Tuple{RangeAxis{Float64, :Lon, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, RangeAxis{Float64, :Lat, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{11, UInt8}, :Distfuns, Vector{Zarr.MaxLengthStrings.MaxLengthString{11, UInt8}}}, CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{6, UInt8}, :FrequencyBins, Vector{Zarr.MaxLengthStrings.MaxLengthString{6, UInt8}}}}, (), NamedTuple{(:palfreqtiny, :palnotiny, :metrics), Tuple{YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}, YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}, YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}}}}; copycols::Bool)
    @ DataFrames ~/.julia/packages/DataFrames/3mEXm/src/other/tables.jl:58
 [17] DataFrame(x::YAXArrays.DAT.CubeIterator{Array{NTuple{4, UnitRange{Int64}}, 4}, Tuple{Matrix{Float32}, Matrix{Float32}, Array{Union{Missing, Float32}, 4}}, Tuple{YAXArrays.YAXTools.PickAxisArray{Float32, 2, Matrix{Float32}, (1, 2), nothing}, YAXArrays.YAXTools.PickAxisArray{Float32, 2, Matrix{Float32}, (1, 2), nothing}, YAXArrays.YAXTools.PickAxisArray{Union{Missing, Float32}, 4, Array{Union{Missing, Float32}, 4}, (1, 2, 3, 4), nothing}}, Tuple{RangeAxis{Float64, :Lon, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, RangeAxis{Float64, :Lat, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{11, UInt8}, :Distfuns, Vector{Zarr.MaxLengthStrings.MaxLengthString{11, UInt8}}}, CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{6, UInt8}, :FrequencyBins, Vector{Zarr.MaxLengthStrings.MaxLengthString{6, UInt8}}}}, (), NamedTuple{(:palfreqtiny, :palnotiny, :metrics), Tuple{YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}, YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}, YAXArrays.DAT.SentinelMissings.SentinelMissing{Float32, NaN32}}}})
    @ DataFrames ~/.julia/packages/DataFrames/3mEXm/src/other/tables.jl:49
 [18] top-level scope
    @ REPL[285]:1

The size of the corresponding cubes have the following axes and sizes:

julia> palfreqtiny
YAXArray with the following dimensions
Lon                 Axis with 8 Elements from 684669.15 to 684879.15
Lat                 Axis with 6 Elements from 9.39391066e6 to 9.39376066e6
Total size: 192.0 bytes


julia> palnotiny
YAXArray with the following dimensions
Lon                 Axis with 8 Elements from 684669.15 to 684879.15
Lat                 Axis with 6 Elements from 9.39391066e6 to 9.39376066e6
Total size: 192.0 bytes


julia> cubetiny
YAXArray with the following dimensions
Distfuns            Axis with 3 elements: euclidean crosscormax cityblock 
FrequencyBins       Axis with 3 elements: Fast Annual Slow 
Lon                 Axis with 8 Elements from 684669.15 to 684879.15
Lat                 Axis with 6 Elements from 9.39391066e6 to 9.39376066e6
Total size: 1.69 KB

If I permute cubetiny so that the Lon Axis is the first and Lat is the second axis, the dataframe generation works.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure at the DataFrame(cttiny) entry point using a CubeTable whose Lon and Lat axes are in positions 3 and 4. Read the YAXArrays.DAT iterator path shown in the stack trace, especially updatear and dciterators.jl, and compare it with the working permuted case. Done means DataFrame conversion works without requiring Lon and Lat to be the first two axes.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.