nullif does not introduce nulls for RunEndEncoded and Union arrays
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
### Describe the bug
`nullif(left, mask)` should return `null` at positions where `mask` is `true`.
The current generic implementation adds a top-level null bitmap to the input `ArrayData`. This works for ordinary arrays, but it does not work for `RunEndEncoded` or `Union` arrays:
- Run-end encoded arrays derive logical nulls from their `values` child array.
- Union arrays derive logical nulls from the selected child value.
- Neither representation uses a top-level null bitmap to represent logical nulls.
As a result, the null bitmap written by `nullif` is ignored when the result is converted back to `RunArray` or `UnionArray`. The operation succeeds but silently returns non-null values where nulls were requested.
### To Reproduce
```rust
use arrow_array::{BooleanArray, Int16Array, Int32Array, RunArray};
use arrow_array::types::Int16Type;
use arrow_select::nullif::nullif;
let ree = RunArray::::try_new(
&Int16Array::from(vec![1, 2]),
&Int32Array::from(vec![10, 20]),
)?;
let mask = BooleanArray::from(vec![Some(false), Some(true)]);
let result = nullif(&ree, &mask)?;
### Expected behavior
[10, null]
### Additional context
[10, 20]
The same issue occurs for UnionArray: a top-level null bitmap is added, but it is ignored by the Union logical-null representation.
Possible direction
nullif should dispatch explicitly for these types instead of using the generic top-level null bitmap path.
- For RunEndEncoded, one possible implementation is to convert mask == true positions into nullable take indices and reuse the Run-End Encoded take implementation.
- For Union, nulls must be represented by selecting a nullable child and writing a null value into that child.
- If a Union has no nullable child field, we need to define whether the kernel should return an error or widen the output field nullability.
Related
- #10909
Contributor guide
Assessment
This issue has not been assessed yet.