Unexpected table schema
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
It appears that Arrow.jl somehow incorrectly handles columns with nullable types (Union with Nothing) as the schema of the resulting table containing such a column doesn't match what's reported by `Tables.schema`.
For example, below is an example with a column of type `Union{Int64,Nothing}` that gets somehow interpreted by Arrow.jl as `Union{Missing, Nothing, Int64}` (should not include Missing).
```julia
julia> using Arrow
julia> tbl = Arrow.Table(Arrow.tobuffer((nothingints=Vector{Union{Int64,Missing}}([1,2,3,missing]),)))
Arrow.Table with 4 rows, 1 columns, and schema:
:nothingints Union{Missing, Int64}
julia> tbl = Arrow.Table(Arrow.tobuffer((nothingints=Vector{Union{Int64,Nothing}}([1,2,3,nothing]),)))
Arrow.Table with 4 rows, 1 columns, and schema:
:nothingints Union{Missing, Nothing, Int64}
julia> using Tables
julia> Tables.schema((nothingints=Vector{Union{Int64,Missing}}([1,2,3,missing]),))
Tables.Schema:
:nothingints Union{Missing, Int64}
julia> Tables.schema((nothingints=Vector{Union{Int64,Nothing}}([1,2,3,nothing]),))
Tables.Schema:
:nothingints Union{Nothing, Int64}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the two Julia examples in the issue and compare Arrow.Table's reported schema with Tables.schema for columns typed as Union{Int64,Nothing}. Trace the schema handling for nullable columns; done means the Arrow table reports Union{Nothing,Int64} without adding Missing, while the existing Missing case remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100