apache / apache/arrow-julia

Deserialization as Vector{SubArray} breaks `push!` on DataFrame

Open
#506 7 comments 1 reaction 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

I'm using Arrow v2.7.2 with DataFrames v1.6.1 on Julia 1.10, and am running into an issue that seems to stem from Arrow.jl deserializing my `Vector{Vector{T}}` columns as `Vector{SubArray{...}}`:

```julia
julia> using Arrow, DataFrames

julia> df = DataFrame(foo=Vector{Int}[]);

julia> push!(df, [[1,2,3]])
1×1 DataFrame
Row │ foo
│ Array…
─────┼───────────
1 │ [1, 2, 3]

julia> Arrow.write("/tmp/test.arrow", df)
"/tmp/test.arrow"

julia> df2 = copy(DataFrame(Arrow.Table("/tmp/test.arrow")));

julia> typeof(df2.foo)
Vector{SubArray{Int64, 1, Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}} (alias for Array{SubArray{Int64, 1, Arrow.Primitive{Int64, Array{Int64, 1}}, Tuple{UnitRange{Int64}}, true}, 1})
```

This breaks certain `push!`es on the dataframe, which I haven't been able to reproduce in isolation, but which looks as follows:

```
MethodError: Cannot `convert` an object of type Vector{Int64} to an object of type SubArray{Int64, 1, Arrow.Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}

Stacktrace:
[1] push!(a::Vector{SubArray{Int64, 1, Arrow.Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}}, item::Vector{Int64})
@ Base ./array.jl:1118
[2] _row_inserter!(df::DataFrame, loc::Int64, row::Tuple{String, Vector{Int64}, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, String, Bool, Bool, Bool, Vector{Int64}, Vector{Int64}, Vector{Int64}, String, String, Float64}, mode::Val{:push}, promote::Bool)
@ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/dataframe/insertion.jl:663
[3] push!(df::DataFrame, row::Tuple{String, Vector{Int64}, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, String, Bool, Bool, Bool, Vector{Int64}, Vector{Int64}, Vector{Int64}, String, String, Float64})
@ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/dataframe/insertion.jl:457
```

It's possible I'm doing something wrong; first time Arrow.jl user here.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the example with Julia 1.10, Arrow 2.7.2, and DataFrames 1.6.1, starting at Arrow.write and Arrow.Table for Vector{Vector{T}} columns. Trace the resulting push! failure through DataFrames' insertion.jl:663 and determine the expected column behavior; done means the reported DataFrame insertion no longer raises the shown MethodError or the expected limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.