apache / apache/arrow-rs

Define nullability semantics for kernels on REE and Union arrays

Open
#10,992 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

### Is your feature request related to a problem or challenge?

This issue follows up on #10909 and accompanies the fix for `nullif` on
RunEndEncoded and Union arrays.

Several compute kernels can introduce logical nulls, for example through null
take indices or a `nullif` condition.

For most Arrow arrays, a top-level validity bitmap represents these nulls.
RunEndEncoded and Union arrays are different:

- RunEndEncoded derives logical nulls from its `values` child.
- Union has no top-level validity bitmap; a logical null must be represented by
a nullable child selected by its type id.

As a result, a kernel can currently produce physical nulls even when the
corresponding field metadata is marked as non-nullable.

### Describe the solution you'd like

A kernel must not silently drop or fabricate requested nulls.

- For `RunEndEncoded`, if the `values` field is non-nullable and an operation
needs to introduce a null, return a compute error.
- For `Union`, a null may be represented using a nullable child field.
If no child field is nullable and an operation needs to introduce a null,
return a compute error.
- Kernels that preserve an existing null should continue to do so without
requiring a new nullable representation.
- This policy should be applied consistently to kernels that can introduce
nulls, including `take`, `nullif`, and similar compute kernels.

### Describe alternatives you've considered

Together with #10909 and the `nullif` fix, this policy keeps the physical
result consistent with its field metadata.

In particular, a field marked as non-nullable must not produce output
containing nulls. Returning an error when a requested null cannot be
represented is preferable to producing a result whose data contradicts its
declared nullability.

### Additional context

PR #10909 fixed handling of null take indices for RunEndEncoded and Union
arrays. This issue tracks the broader nullability contract and its consistent
application across compute kernels.

This issue tracks the broader nullability contract for REE and Union arrays and
the consistent application of that contract across compute kernels.

Contributor guide

Open the contributing guide

Research direction

Start by reading PR #10909 and the compute-kernel implementations for take and nullif, then identify other kernels that can introduce nulls for RunEndEncoded and Union arrays. The work is done when requested nulls are preserved or represented consistently, and kernels return compute errors when the field metadata provides no valid representation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.