Avoid forced copy in `Array::logical_nulls`
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Some code requires "logical nullability" which is distinct from the "physical" nullability (what is defined by an Arrow arrays NullBuffer, if it has one)
To this end, arrow includes `Array::logical_nulls` which will compute logical nulls.
The issue is that it also potentially copies the `NullBuffer` for arrays that have it pre-computed, such as `PrimitiveArray` and `StringArray`. While the copy is perhaps not that bad (it is several `Arc`s) I would like to avoid it entirely if possible
**Describe the solution you'd like**
I would like to change `logical_nulls` to return a `Cow` to avoid the copy, as I propose in https://github.com/coralogix/arrow-datafusion/pull/221
**Describe alternatives you've considered**
We can not make any changes
**Additional context**
Thus came up in DataFusion PR https://github.com/apache/arrow-datafusion/pull/8511
Logical nulls was added in https://github.com/apache/arrow-rs/pull/4691
Contributor guide
Research direction
Start at Array::logical_nulls and review the proposed Cow approach in coralogix/arrow-datafusion#221, along with the DataFusion context in PR #8511. Trace the existing callers and API implications; the work is done when precomputed null buffers avoid the forced copy while logical nulls still behave correctly for other arrays.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100