apache / apache/arrow-rs

[Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible

Open
#10,245 19 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

### Is your feature request related to a problem or challenge?

As @Jefffrey has observed in https://github.com/apache/arrow-rs/pull/10229#discussion_r3496012625, switching from `OffsetBufferBuilder` / `BufferBuilder` to use `Vec<..>` often results in a non trivial speedup.

This is likely because the Rust language has a *highly* optimized Vec implementation

### Describe the solution you'd like

Basically try and change any code like this

```rust
let mut values = BufferBuilder::::new(len);
```

To use
```rust
let mut values = Vec::with_capacity(len);
```

And various equivalent

## Note to contributors:
please make targeted PRs (that change one or two usages each) to keep the PRs easy to review and thus fast to merge

## Remaining callsites

(as of [`0e581556`](https://github.com/apache/arrow-rs/commit/0e581556e8ba4ec4fff4d1e0ed11d44730b92767))

- [ ] [`arrow-arith/src/arity.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-arith/src/arity.rs): [`try_binary` nullable path](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-arith/src/arity.rs#L285) builds the output values with `BufferBuilder::` (likely covered by #10518)
- [x] [`arrow-cast/src/cast/mod.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-cast/src/cast/mod.rs): [`cast_byte_container`](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-cast/src/cast/mod.rs#L2835) converts offsets between offset types with `BufferBuilder::` (done in #10867)
- [x] [`arrow-ipc/src/writer.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-ipc/src/writer.rs): [`into_zero_offset_run_array`](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-ipc/src/writer.rs#L1295) re-encodes sliced run-ends with `BufferBuilder::` (done in #11005)
- [x] [`arrow-row/src/variable.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-row/src/variable.rs): [`decode_binary`](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-row/src/variable.rs#L287) builds offsets with `BufferBuilder::` (done in #10851)
- [ ] [`arrow-select/src/zip.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-select/src/zip.rs): [byte-array zip path](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-select/src/zip.rs#L562) uses `OffsetBufferBuilder::` (PR open: #10913)
- [x] ~[`arrow-avro/src/reader/record.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-avro/src/reader/record.rs): [`Decoder` enum variants](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-avro/src/reader/record.rs#L245) (`Binary`/`String`/`StringView`/`Array`/`Map`) hold `OffsetBufferBuilder` as long-lived decoder state (~15 sites; a larger refactor than the others since the builders are not per-call scratch buffers)~ won't do: per discussion in #10923, keeping `OffsetBufferBuilder` here is clearer and the usages are not in a tight hot loop

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the unchecked callsites in arrow-arith/src/arity.rs and arrow-select/src/zip.rs, especially try_binary and the byte-array zip path. Compare the completed conversions listed in the issue, then verify that each targeted builder usage is replaced where appropriate and the relevant crate tests pass; the zip work already has PR #10913 open.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.