apache / apache/arrow-rs

Avoid forced copy in `Array::logical_nulls`

Open
#5,208 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Some code requires "logical nullability" which is distinct from the "physical" nullability (what is defined by an Arrow arrays NullBuffer, if it has one)

To this end, arrow includes `Array::logical_nulls` which will compute logical nulls.

The issue is that it also potentially copies the `NullBuffer` for arrays that have it pre-computed, such as `PrimitiveArray` and `StringArray`. While the copy is perhaps not that bad (it is several `Arc`s) I would like to avoid it entirely if possible

**Describe the solution you'd like**
I would like to change `logical_nulls` to return a `Cow` to avoid the copy, as I propose in https://github.com/coralogix/arrow-datafusion/pull/221

**Describe alternatives you've considered**
We can not make any changes

**Additional context**
Thus came up in DataFusion PR https://github.com/apache/arrow-datafusion/pull/8511

Logical nulls was added in https://github.com/apache/arrow-rs/pull/4691

Contributor guide

Open the contributing guide

Research direction

Start at Array::logical_nulls and review the proposed Cow approach in coralogix/arrow-datafusion#221, along with the DataFusion context in PR #8511. Trace the existing callers and API implications; the work is done when precomputed null buffers avoid the forced copy while logical nulls still behave correctly for other arrays.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.