apache / apache/arrow-julia

Missing values are not handled when converting `DictEncoded` to `PooledArray` via `copy`

Open
#220 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

When copying a column of type `DictEncoded`, the missing values in it are not handled. The resulting `PooledArray` does not have `missing` in its `pool`. Because of that, if one tries to access the element that is supposed to be `missing`, an `UndefRefError` is raised.

Here is an illustration with an example data file:
[cat_with_missing.feather.zip](https://github.com/JuliaData/Arrow.jl/files/6743124/cat_with_missing.feather.zip)

```julia
using Arrow
tb = Arrow.Table("cat_with_missing.feather")
```
```julia
julia> tb.A
3-element Arrow.DictEncoded{Union{Missing, String}, Int8, Arrow.List{Union{Missing, String}, Int32, Vector{UInt8}}}:
missing
"a"
"b"

julia> tb.A[1]
missing

julia> A = copy(tb.A)
3-element PooledArrays.PooledVector{Union{Missing, String}, Int8, Vector{Int8}}:
#undef
"a"
"b"

julia> A[1]
ERROR: UndefRefError: access to undefined reference
Stacktrace:
[1] getindex(A::PooledArrays.PooledVector{Union{Missing, String}, Int8, Vector{Int8}}, I::Int64)
@ PooledArrays ~/.julia/packages/PooledArrays/CV8kA/src/PooledArrays.jl:451
...
```
The current implementation of `copy` directly uses the encoding data as the `pool`, but that `pool` does not contain `missing`:
```julia
julia> tb.A.encoding.data
2-element Arrow.List{Union{Missing, String}, Int32, Vector{UInt8}}:
"a"
"b"
```
This seems to be related to ongoing work on changing how the `missing` values should be represented in a `PooledArray`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the copy implementation for Arrow.DictEncoded and inspect how encoding.data is passed into the PooledArray pool. Reproduce the provided missing-value example, then verify that the pool represents missing and that accessing the copied column's missing element no longer raises UndefRefError.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.