Nullable fields don't always need Union{Missing, T}
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
I'm trying to implement the GeoArrow spec, which gives back coordinates in a deeply nested list of a FixedList (a point). Because these lists are theoretically nullable, in Julia we get an deeply nested list with `Union`s of `Missing`, even though these vectors contain no `missing`s. An example for a column of LineStrings (there are geometry types that require two more levels of nesting):
```2-element Arrow.List{Vector{Union{Missing, Vector{Union{Missing, Tuple{Float64, Float64}}}}}```
It's pretty hard to convert these elements to a concrete `Vector{Vector{NTuple, Float64}}` without allocating. Is there a way to edit the view to be non missing? An alternative way would be to pass `all(validitybitmap)` in `build` to `juliaeltype`, so we only set Missing when there are actual missing values.
I'm happy to make a PR if there's consensus on what to do.
Might be related to #373.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the build path and juliaeltype, especially how validitybitmap is handled; issue #373 may provide related context. Decide whether non-missing nested views should be supported or whether Missing should depend on all(validitybitmap), then confirm the chosen behavior with coverage for nested LineStrings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100