(de)serialization behavior of `missing`/`nothing`
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
In Julia, there is (generally) a useful/meaningful semantic distinction between `nothing` and `missing`. IIUC, Arrow doesn't really have equivalent values that capture this distinction, but instead has `null` which might be used for either. This results in a bit of an impedance mismatch for us to resolve when (de)serializing `nothing`/`missing` data.
The current behavior feels like it "resolves" the impedance mismatch just by tossing this information altogether and normalizing to a single value, but the value it chooses to normalize to feels weird to me:
```
julia> Arrow.Table(Arrow.tobuffer((x = [missing, missing],))).x
2-element Arrow.NullVector{Missing}:
missing
missing
julia> Arrow.Table(Arrow.tobuffer((x = [nothing, nothing],))).x
2-element Arrow.NullVector{Nothing}:
nothing
nothing
julia> Arrow.Table(Arrow.tobuffer((x = [nothing, missing],))).x
2-element Arrow.NullVector{Nothing}:
nothing
nothing
julia> Arrow.Table(Arrow.tobuffer((x = Any[nothing, missing],))).x
2-element Arrow.NullVector{Missing}:
missing
missing
```
It seems to me like Arrow.jl should either:
1. find some way to consistently preserve this distinction in all cases when (de)serializing Julia data (e.g. so that `[nothing, missing]` would roundtrip as `[nothing, missing]`)
2. lean all-in on dropping the distinction, and force callers to pick what they want to interpret incoming Arrow `null`s (e.g. `nothing` or `missing`) at read time.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Arrow.Table(Arrow.tobuffer(...)) examples in the issue for vectors containing missing and nothing, including the mixed Any vector. Trace the serialization and deserialization entry points involved, then determine which of the two proposed semantics is intended. Done means the chosen behavior is consistent and the examples demonstrate the resulting round-trip or read-time interpretation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100