apache / apache/arrow-julia

Unexpected table schema

Open
#283 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

It appears that Arrow.jl somehow incorrectly handles columns with nullable types (Union with Nothing) as the schema of the resulting table containing such a column doesn't match what's reported by `Tables.schema`.

For example, below is an example with a column of type `Union{Int64,Nothing}` that gets somehow interpreted by Arrow.jl as `Union{Missing, Nothing, Int64}` (should not include Missing).

```julia
julia> using Arrow

julia> tbl = Arrow.Table(Arrow.tobuffer((nothingints=Vector{Union{Int64,Missing}}([1,2,3,missing]),)))
Arrow.Table with 4 rows, 1 columns, and schema:
:nothingints Union{Missing, Int64}

julia> tbl = Arrow.Table(Arrow.tobuffer((nothingints=Vector{Union{Int64,Nothing}}([1,2,3,nothing]),)))
Arrow.Table with 4 rows, 1 columns, and schema:
:nothingints Union{Missing, Nothing, Int64}

julia> using Tables

julia> Tables.schema((nothingints=Vector{Union{Int64,Missing}}([1,2,3,missing]),))
Tables.Schema:
:nothingints Union{Missing, Int64}

julia> Tables.schema((nothingints=Vector{Union{Int64,Nothing}}([1,2,3,nothing]),))
Tables.Schema:
:nothingints Union{Nothing, Int64}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the two Julia examples in the issue and compare Arrow.Table's reported schema with Tables.schema for columns typed as Union{Int64,Nothing}. Trace the schema handling for nullable columns; done means the Arrow table reports Union{Nothing,Int64} without adding Missing, while the existing Missing case remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.