apache / apache/arrow-julia

Nullable fields don't always need Union{Missing, T}

Open
#384 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

I'm trying to implement the GeoArrow spec, which gives back coordinates in a deeply nested list of a FixedList (a point). Because these lists are theoretically nullable, in Julia we get an deeply nested list with `Union`s of `Missing`, even though these vectors contain no `missing`s. An example for a column of LineStrings (there are geometry types that require two more levels of nesting):

```2-element Arrow.List{Vector{Union{Missing, Vector{Union{Missing, Tuple{Float64, Float64}}}}}```

It's pretty hard to convert these elements to a concrete `Vector{Vector{NTuple, Float64}}` without allocating. Is there a way to edit the view to be non missing? An alternative way would be to pass `all(validitybitmap)` in `build` to `juliaeltype`, so we only set Missing when there are actual missing values.

I'm happy to make a PR if there's consensus on what to do.

Might be related to #373.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the build path and juliaeltype, especially how validitybitmap is handled; issue #373 may provide related context. Decide whether non-missing nested views should be supported or whether Missing should depend on all(validitybitmap), then confirm the chosen behavior with coverage for nested LineStrings.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.