apache / apache/arrow-rs

Safe API to replace `NullBuffers` for Arrays

Open
#6,528 20 comments 0 reactions 0 assignees View on GitHub
arrow enhancement help wanted
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
While implementing https://github.com/apache/datafusion/pull/12792 and various other things in DataFusion I find myself often wanting to combine a filter and a null mask

The relevant code is like

```rust
// combine existing nulls, if any, and a filter:
let nulls = filtered_null_mask(opt_filter, input);

// make a new array with a new null buffer
let output: ArrayRef = match input.data_type() {
// TODO it would be nice to have safe apis in arrow-rs to update the null buffers in the arrays
DataType::Utf8 => {
let input = input.as_string::();
// safety: values / offsets came from a valid string array, so are valid utf8
// and we checked nulls has the same length as values
unsafe {
Arc::new(StringArray::new_unchecked(
input.offsets().clone(),
input.values().clone(),
nulls,
))
}
}
```

**Describe the solution you'd like**
I would like an API like `with_nulls` that returns a new array with the same data but a new null mask so my code would look like

```rust
// combine existing nulls, if any, and a filter:
let nulls = filtered_null_mask(opt_filter, input);

// make a new array, with the same data but a new null buffer
// panics if the nulls length doesn't match the array's length
let output = input.with_nulls(nulls);
```

**Describe alternatives you've considered**
I can keep using the unsafe APIs

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the ArrayRef interface and the concrete array APIs around StringArray::new_unchecked, offsets, values, and null buffers. Determine how a safe with_nulls operation can preserve existing array data while validating the null mask length, then check the existing array tests and add coverage for the requested behavior across relevant array types.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.