apache / apache/arrow-julia

Type instability in getcolumn

Open
#499 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

In `Arrow.Table` all columns are stored in a `Vector{AbstractVector}`. This causes downstream type instability problems and performance problems when iterating over a single column.

```julia
julia> using Arrow, Tables

julia> buf = Arrow.tobuffer((a=[1,2,3], b=[4,5,6]));

julia> tt = Arrow.Table(buf)
Arrow.Table with 3 rows, 2 columns, and schema:
:a Int64
:b Int64

julia> @code_warntype Tables.getcolumn(tt, :a)
MethodInstance for Tables.getcolumn(::Arrow.Table, ::Symbol)
from getcolumn(t::Arrow.Table, nm::Symbol) @ Arrow ~/.julia/packages/Arrow/ID4np/src/table.jl:369
Arguments
#self#::Core.Const(Tables.getcolumn)
t::Arrow.Table
nm::Symbol
Body::AbstractVector
1 ─ %1 = Arrow.lookup(t)::Dict{Symbol, AbstractVector}
│ %2 = Base.getindex(%1, nm)::AbstractVector
└── return %2
```

This uses Julia v1.10 and Arrow v2.7.1.

```julia
julia> versioninfo()
Julia Version 1.10.0
Commit 3120989f39b (2023-12-25 18:01 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 48 × Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, skylake-avx512)
Threads: 5 on 48 virtual cores
Environment:
JULIA_NUM_THREADS = 4
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/table.jl at the getcolumn method around line 369, then inspect Arrow.lookup and the Vector{AbstractVector} column storage described in the issue. Reproduce the Julia 1.10 example and run @code_warntype on Tables.getcolumn(tt, :a); the issue is done when single-column access no longer exhibits the reported abstract return type and associated instability.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.