Missing values are not handled when converting `DictEncoded` to `PooledArray` via `copy`
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
When copying a column of type `DictEncoded`, the missing values in it are not handled. The resulting `PooledArray` does not have `missing` in its `pool`. Because of that, if one tries to access the element that is supposed to be `missing`, an `UndefRefError` is raised.
Here is an illustration with an example data file:
[cat_with_missing.feather.zip](https://github.com/JuliaData/Arrow.jl/files/6743124/cat_with_missing.feather.zip)
```julia
using Arrow
tb = Arrow.Table("cat_with_missing.feather")
```
```julia
julia> tb.A
3-element Arrow.DictEncoded{Union{Missing, String}, Int8, Arrow.List{Union{Missing, String}, Int32, Vector{UInt8}}}:
missing
"a"
"b"
julia> tb.A[1]
missing
julia> A = copy(tb.A)
3-element PooledArrays.PooledVector{Union{Missing, String}, Int8, Vector{Int8}}:
#undef
"a"
"b"
julia> A[1]
ERROR: UndefRefError: access to undefined reference
Stacktrace:
[1] getindex(A::PooledArrays.PooledVector{Union{Missing, String}, Int8, Vector{Int8}}, I::Int64)
@ PooledArrays ~/.julia/packages/PooledArrays/CV8kA/src/PooledArrays.jl:451
...
```
The current implementation of `copy` directly uses the encoding data as the `pool`, but that `pool` does not contain `missing`:
```julia
julia> tb.A.encoding.data
2-element Arrow.List{Union{Missing, String}, Int32, Vector{UInt8}}:
"a"
"b"
```
This seems to be related to ongoing work on changing how the `missing` values should be represented in a `PooledArray`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the copy implementation for Arrow.DictEncoded and inspect how encoding.data is passed into the PooledArray pool. Reproduce the provided missing-value example, then verify that the pool represents missing and that accessing the copied column's missing element no longer raises UndefRefError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100