Define nullability semantics for kernels on REE and Union arrays
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
### Is your feature request related to a problem or challenge?
This issue follows up on #10909 and accompanies the fix for `nullif` on
RunEndEncoded and Union arrays.
Several compute kernels can introduce logical nulls, for example through null
take indices or a `nullif` condition.
For most Arrow arrays, a top-level validity bitmap represents these nulls.
RunEndEncoded and Union arrays are different:
- RunEndEncoded derives logical nulls from its `values` child.
- Union has no top-level validity bitmap; a logical null must be represented by
a nullable child selected by its type id.
As a result, a kernel can currently produce physical nulls even when the
corresponding field metadata is marked as non-nullable.
### Describe the solution you'd like
A kernel must not silently drop or fabricate requested nulls.
- For `RunEndEncoded`, if the `values` field is non-nullable and an operation
needs to introduce a null, return a compute error.
- For `Union`, a null may be represented using a nullable child field.
If no child field is nullable and an operation needs to introduce a null,
return a compute error.
- Kernels that preserve an existing null should continue to do so without
requiring a new nullable representation.
- This policy should be applied consistently to kernels that can introduce
nulls, including `take`, `nullif`, and similar compute kernels.
### Describe alternatives you've considered
Together with #10909 and the `nullif` fix, this policy keeps the physical
result consistent with its field metadata.
In particular, a field marked as non-nullable must not produce output
containing nulls. Returning an error when a requested null cannot be
represented is preferable to producing a result whose data contradicts its
declared nullability.
### Additional context
PR #10909 fixed handling of null take indices for RunEndEncoded and Union
arrays. This issue tracks the broader nullability contract and its consistent
application across compute kernels.
This issue tracks the broader nullability contract for REE and Union arrays and
the consistent application of that contract across compute kernels.
Contributor guide
Research direction
Start by reading PR #10909 and the compute-kernel implementations for take and nullif, then identify other kernels that can introduce nulls for RunEndEncoded and Union arrays. The work is done when requested nulls are preserved or represented consistently, and kernels return compute errors when the field metadata provides no valid representation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100