apache / apache/arrow-julia

DataAPI.jl integration questions

Open
#193 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

I have several questions regarding DataAPI.jl integration of Arrow.jl. They are mostly stemming from the fact that I do not know the details of Arrow.jl implementation so I might be asking about something obvious:

1. Why does `DataAPI.refpool(x::DictEncoded) = copy(x.encoding.data)` perform a copy? The question is following the fact that this will negatively affect the performance of `groupby` and `join*` in DataFrames.jl
2. Similarly we now have `DataAPI.refarray(x::DictEncoded{T, S}) where {T, S} = x.indices .+ one(S)` which allocates. In other packages (CategoricalArrays.jl, PooledArrays.jl) we have an implementation that does not allocate (again - allocation will negatively affect performance)
3. Why in `DataAPI.levels(x::DictEncoded)` we do not try sorting the levels? Also it seems that instead of `deleteat!(rp, ismissing.(rp))` we could just use `collect` over `skipmissing` wrapper (and this combined with non-copying `rp = DataAPI.refpool(x)` as suggested above should lend a faster implementation).

Thank you!

Contributor guide

No contributing guide indexed for this repository

Research direction

Read the DictEncoded implementations of DataAPI.refpool, refarray, and levels, then compare their allocation behavior with the CategoricalArrays.jl and PooledArrays.jl implementations named in the issue. Resolve whether the copying, index conversion, level ordering, and missing-value handling are intentional; done requires a maintainer decision and, if changes are accepted, coverage of the resulting behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.