DataAPI.jl integration questions
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
I have several questions regarding DataAPI.jl integration of Arrow.jl. They are mostly stemming from the fact that I do not know the details of Arrow.jl implementation so I might be asking about something obvious:
1. Why does `DataAPI.refpool(x::DictEncoded) = copy(x.encoding.data)` perform a copy? The question is following the fact that this will negatively affect the performance of `groupby` and `join*` in DataFrames.jl
2. Similarly we now have `DataAPI.refarray(x::DictEncoded{T, S}) where {T, S} = x.indices .+ one(S)` which allocates. In other packages (CategoricalArrays.jl, PooledArrays.jl) we have an implementation that does not allocate (again - allocation will negatively affect performance)
3. Why in `DataAPI.levels(x::DictEncoded)` we do not try sorting the levels? Also it seems that instead of `deleteat!(rp, ismissing.(rp))` we could just use `collect` over `skipmissing` wrapper (and this combined with non-copying `rp = DataAPI.refpool(x)` as suggested above should lend a faster implementation).
Thank you!
Contributor guide
No contributing guide indexed for this repository
Research direction
Read the DictEncoded implementations of DataAPI.refpool, refarray, and levels, then compare their allocation behavior with the CategoricalArrays.jl and PooledArrays.jl implementations named in the issue. Resolve whether the copying, index conversion, level ordering, and missing-value handling are intentional; done requires a maintainer decision and, if changes are accepted, coverage of the resulting behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100