Safe API to replace `NullBuffers` for Arrays
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
While implementing https://github.com/apache/datafusion/pull/12792 and various other things in DataFusion I find myself often wanting to combine a filter and a null mask
The relevant code is like
```rust
// combine existing nulls, if any, and a filter:
let nulls = filtered_null_mask(opt_filter, input);
// make a new array with a new null buffer
let output: ArrayRef = match input.data_type() {
// TODO it would be nice to have safe apis in arrow-rs to update the null buffers in the arrays
DataType::Utf8 => {
let input = input.as_string::();
// safety: values / offsets came from a valid string array, so are valid utf8
// and we checked nulls has the same length as values
unsafe {
Arc::new(StringArray::new_unchecked(
input.offsets().clone(),
input.values().clone(),
nulls,
))
}
}
```
**Describe the solution you'd like**
I would like an API like `with_nulls` that returns a new array with the same data but a new null mask so my code would look like
```rust
// combine existing nulls, if any, and a filter:
let nulls = filtered_null_mask(opt_filter, input);
// make a new array, with the same data but a new null buffer
// panics if the nulls length doesn't match the array's length
let output = input.with_nulls(nulls);
```
**Describe alternatives you've considered**
I can keep using the unsafe APIs
**Additional context**
Contributor guide
Research direction
Start by inspecting the ArrayRef interface and the concrete array APIs around StringArray::new_unchecked, offsets, values, and null buffers. Determine how a safe with_nulls operation can preserve existing array data while validating the null mask length, then check the existing array tests and add coverage for the requested behavior across relevant array types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100