apache / apache/arrow-rs

RunArray::slice() should align run_ends and values with the logical slice

Open
#10,017 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

## `RunArray::slice()` should trim the underlying values array

`RunArray::slice()` currently tracks slicing logically without adjusting the underlying values array, so a sliced `RunArray` still references the full original values. This has performance implications for patterns that operate on `array.values()`.

### Example

```rust
let run = Int32Array::from(vec![3, 6, 9]);
let values = Int32Array::from(vec![1, 2, 3]);
// [1, 1, 1, 2, 2, 2, 3, 3, 3]
let array = RunArray::try_new(&run, &values).unwrap();

// Logically [2, 2, 2], but still references the original arrays:
// {1, 1, 1, [2, 2, 2], 3, 3, 3}
let array_sliced = array.slice(3, 3);
```

Ideally `array_sliced` would be `{ run_ends: [3], values: [2] }`, but the [current implementation of `RunArray::slice()`](https://github.com/apache/arrow-rs/blob/8acab7b5371470deff6c211899295d2bb3030dfc/arrow-array/src/array/run_array.rs#L344) preserves the full values array.

### Why it matters

Patterns like the following do extra work on values outside the slice:

```rust
let values = date_part(array.values(), part)?;
let new_array = array.with_values(values);
```

Not a correctness bug, but worth avoiding.

### Regression checks

Each call site of `RunArray::slice()` needs to be reviewed to ensure this change doesn't introduce breaking behavior.

### Reference

[[Original comment thread](https://github.com/apache/arrow-rs/pull/9959#discussion_r3270678608)](https://github.com/apache/arrow-rs/pull/9959#discussion_r3270678608)

Contributor guide

Open the contributing guide

Research direction

Start in arrow-array/src/array/run_array.rs at the current RunArray::slice() implementation, then review each call site mentioned in the issue for assumptions about the underlying arrays. Verify that slicing produces aligned run_ends and values, such as run_ends [3] and values [2] for the example, and check the relevant regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.