apache / apache/arrow-rs

nullif does not introduce nulls for RunEndEncoded and Union arrays

Open
#10,990 1 comment 0 reactions 1 assignee Claimed by @yongster View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Describe the bug

`nullif(left, mask)` should return `null` at positions where `mask` is `true`.

The current generic implementation adds a top-level null bitmap to the input `ArrayData`. This works for ordinary arrays, but it does not work for `RunEndEncoded` or `Union` arrays:

- Run-end encoded arrays derive logical nulls from their `values` child array.
- Union arrays derive logical nulls from the selected child value.
- Neither representation uses a top-level null bitmap to represent logical nulls.

As a result, the null bitmap written by `nullif` is ignored when the result is converted back to `RunArray` or `UnionArray`. The operation succeeds but silently returns non-null values where nulls were requested.

### To Reproduce

```rust
use arrow_array::{BooleanArray, Int16Array, Int32Array, RunArray};
use arrow_array::types::Int16Type;
use arrow_select::nullif::nullif;

let ree = RunArray::::try_new(
&Int16Array::from(vec![1, 2]),
&Int32Array::from(vec![10, 20]),
)?;

let mask = BooleanArray::from(vec![Some(false), Some(true)]);
let result = nullif(&ree, &mask)?;

### Expected behavior

[10, null]

### Additional context

[10, 20]

The same issue occurs for UnionArray: a top-level null bitmap is added, but it is ignored by the Union logical-null representation.
Possible direction
nullif should dispatch explicitly for these types instead of using the generic top-level null bitmap path.
- For RunEndEncoded, one possible implementation is to convert mask == true positions into nullable take indices and reuse the Run-End Encoded take implementation.
- For Union, nulls must be represented by selecting a nullable child and writing a null value into that child.
- If a Union has no nullable child field, we need to define whether the kernel should return an error or widen the output field nullability.
Related
- #10909

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.