Addtional `Missing` gets injected into Schema
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
In the following example, the column doesn't have `Missing` but after a round trip through Arrow it gained Missing in the Union
```julia
julia> rnt.vector_variant_int64_string
5-element RNTupleField{Vector{Union{Int64, String}}}:
Union{Int64, String}["one"]
Union{Int64, String}["one", 2]
Union{Int64, String}["one", 2, 3]
Union{Int64, String}["one", 2, 3, 4]
Union{Int64, String}["one", 2, 3, 4, 5]
julia> DataFrame(rnt).vector_variant_int64_string
5-element Vector{Vector{Union{Int64, String}}}:
["one"]
["one", 2]
["one", 2, 3]
["one", 2, 3, 4]
["one", 2, 3, 4, 5]
julia> Arrow.write(a, DataFrame(rnt))
"/tmp/jl_Lk5W1G92XO"
julia> Arrow.Table(a)
Arrow.Table with 5 rows, 13 columns, and schema:
:string String
:vector_int32 Vector{Int32} (alias for Array{Int32, 1})
:array_float Vector{Float32} (alias for Array{Float32, 1})
:vector_vector_int32 Vector{Vector{Int32}} (alias for Array{Array{Int32, 1}, 1})
:vector_string Vector{String} (alias for Array{String, 1})
:vector_vector_string Vector{Vector{String}} (alias for Array{Array{String, 1}, 1})
:variant_int32_string Union{Missing, Int32, String}
:vector_variant_int64_string Vector{Union{Missing, Int64, String}} (alias for Array{Union{Missing, Int64, String}, 1})
:tuple_int32_string NamedTuple{(:_0, :_1), Tuple{Int32, String}}
:pair_int32_string NamedTuple{(:_0, :_1), Tuple{Int32, String}}
:vector_tuple_int32_string Vector{NamedTuple{(:_0, :_1), Tuple{Int32, String}}} (alias for Array{NamedTuple{(:_0, :_1), Tuple{Int32, String}}, 1})
:lorentz_vector NamedTuple{(:pt, :eta, :phi, :mass), NTuple{4, Float32}}
:array_lv Vector{NamedTuple{(:pt, :eta, :phi, :mass), NTuple{4, Float32}}} (alias for Array{NamedTuple{(:pt, :eta, :phi, :mass), NTuple{4, Float32}}, 1})
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the Julia REPL round trip using DataFrame, Arrow.write, and Arrow.Table shown in the issue, focusing on vector_variant_int64_string. Trace the schema conversion that changes Union{Int64, String} to Union{Missing, Int64, String}; done means the round-tripped column preserves its original union without an unnecessary Missing type.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100