apache / apache/arrow-julia

Addtional `Missing` gets injected into Schema

Open
#373 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

In the following example, the column doesn't have `Missing` but after a round trip through Arrow it gained Missing in the Union

```julia
julia> rnt.vector_variant_int64_string
5-element RNTupleField{Vector{Union{Int64, String}}}:
Union{Int64, String}["one"]
Union{Int64, String}["one", 2]
Union{Int64, String}["one", 2, 3]
Union{Int64, String}["one", 2, 3, 4]
Union{Int64, String}["one", 2, 3, 4, 5]

julia> DataFrame(rnt).vector_variant_int64_string
5-element Vector{Vector{Union{Int64, String}}}:
["one"]
["one", 2]
["one", 2, 3]
["one", 2, 3, 4]
["one", 2, 3, 4, 5]

julia> Arrow.write(a, DataFrame(rnt))
"/tmp/jl_Lk5W1G92XO"

julia> Arrow.Table(a)
Arrow.Table with 5 rows, 13 columns, and schema:
:string String
:vector_int32 Vector{Int32} (alias for Array{Int32, 1})
:array_float Vector{Float32} (alias for Array{Float32, 1})
:vector_vector_int32 Vector{Vector{Int32}} (alias for Array{Array{Int32, 1}, 1})
:vector_string Vector{String} (alias for Array{String, 1})
:vector_vector_string Vector{Vector{String}} (alias for Array{Array{String, 1}, 1})
:variant_int32_string Union{Missing, Int32, String}
:vector_variant_int64_string Vector{Union{Missing, Int64, String}} (alias for Array{Union{Missing, Int64, String}, 1})
:tuple_int32_string NamedTuple{(:_0, :_1), Tuple{Int32, String}}
:pair_int32_string NamedTuple{(:_0, :_1), Tuple{Int32, String}}
:vector_tuple_int32_string Vector{NamedTuple{(:_0, :_1), Tuple{Int32, String}}} (alias for Array{NamedTuple{(:_0, :_1), Tuple{Int32, String}}, 1})
:lorentz_vector NamedTuple{(:pt, :eta, :phi, :mass), NTuple{4, Float32}}
:array_lv Vector{NamedTuple{(:pt, :eta, :phi, :mass), NTuple{4, Float32}}} (alias for Array{NamedTuple{(:pt, :eta, :phi, :mass), NTuple{4, Float32}}, 1})
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Julia REPL round trip using DataFrame, Arrow.write, and Arrow.Table shown in the issue, focusing on vector_variant_int64_string. Trace the schema conversion that changes Union{Int64, String} to Union{Missing, Int64, String}; done means the round-tripped column preserves its original union without an unnecessary Missing type.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.