apache / apache/arrow-rs

Add API to avoid allocations when using `ArrayData::slice()` on owned ArrayData

Open
#9,140 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

> we can probably avoid an additional allocation for sliced arrays by making a version of `slice()` that consumes self -- like `sliced()` perhaps 🤔

_Originally posted by @alamb in [#9114](https://github.com/apache/arrow-rs/pull/9114/changes#r2670379840)_

The idea is that calling `ArrayData::slice()` creates a *new* array data (needs to allocate a Vec for the new child buffers). For some cases, the caller owns the ArrayData and thus could simply reuse the allocation by taking it as owned

Something like
```rust
impl ArrayData {
/// Like Self::sliced, but takes an owned self, and reuses the allocations, making it more efficient
fn sliced(self, offset: usize, len: usize) -> Self {
...
}
}
```

Contributor guide

Open the contributing guide

Research direction

Start at the ArrayData::slice() implementation and trace how slicing constructs child buffers and allocates its Vec. Define the owned-self slicing API described in the issue, preserving the existing slice behavior while reusing allocations where possible. Done means callers can slice an owned ArrayData without the additional allocation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.