apache / apache/arrow-julia

(de)serialization behavior of `missing`/`nothing`

Open
#258 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

In Julia, there is (generally) a useful/meaningful semantic distinction between `nothing` and `missing`. IIUC, Arrow doesn't really have equivalent values that capture this distinction, but instead has `null` which might be used for either. This results in a bit of an impedance mismatch for us to resolve when (de)serializing `nothing`/`missing` data.

The current behavior feels like it "resolves" the impedance mismatch just by tossing this information altogether and normalizing to a single value, but the value it chooses to normalize to feels weird to me:

```
julia> Arrow.Table(Arrow.tobuffer((x = [missing, missing],))).x
2-element Arrow.NullVector{Missing}:
missing
missing

julia> Arrow.Table(Arrow.tobuffer((x = [nothing, nothing],))).x
2-element Arrow.NullVector{Nothing}:
nothing
nothing

julia> Arrow.Table(Arrow.tobuffer((x = [nothing, missing],))).x
2-element Arrow.NullVector{Nothing}:
nothing
nothing

julia> Arrow.Table(Arrow.tobuffer((x = Any[nothing, missing],))).x
2-element Arrow.NullVector{Missing}:
missing
missing
```

It seems to me like Arrow.jl should either:

1. find some way to consistently preserve this distinction in all cases when (de)serializing Julia data (e.g. so that `[nothing, missing]` would roundtrip as `[nothing, missing]`)
2. lean all-in on dropping the distinction, and force callers to pick what they want to interpret incoming Arrow `null`s (e.g. `nothing` or `missing`) at read time.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Arrow.Table(Arrow.tobuffer(...)) examples in the issue for vectors containing missing and nothing, including the mixed Any vector. Trace the serialization and deserialization entry points involved, then determine which of the two proposed semantics is intended. Done means the chosen behavior is consistent and the examples demonstrate the resulting round-trip or read-time interpretation.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.