[Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
### Is your feature request related to a problem or challenge?
As @Jefffrey has observed in https://github.com/apache/arrow-rs/pull/10229#discussion_r3496012625, switching from `OffsetBufferBuilder` / `BufferBuilder` to use `Vec<..>` often results in a non trivial speedup.
This is likely because the Rust language has a *highly* optimized Vec implementation
### Describe the solution you'd like
Basically try and change any code like this
```rust
let mut values = BufferBuilder::::new(len);
```
To use
```rust
let mut values = Vec::with_capacity(len);
```
And various equivalent
## Note to contributors:
please make targeted PRs (that change one or two usages each) to keep the PRs easy to review and thus fast to merge
## Remaining callsites
(as of [`0e581556`](https://github.com/apache/arrow-rs/commit/0e581556e8ba4ec4fff4d1e0ed11d44730b92767))
- [ ] [`arrow-arith/src/arity.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-arith/src/arity.rs): [`try_binary` nullable path](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-arith/src/arity.rs#L285) builds the output values with `BufferBuilder::` (likely covered by #10518)
- [x] [`arrow-cast/src/cast/mod.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-cast/src/cast/mod.rs): [`cast_byte_container`](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-cast/src/cast/mod.rs#L2835) converts offsets between offset types with `BufferBuilder::` (done in #10867)
- [x] [`arrow-ipc/src/writer.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-ipc/src/writer.rs): [`into_zero_offset_run_array`](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-ipc/src/writer.rs#L1295) re-encodes sliced run-ends with `BufferBuilder::` (done in #11005)
- [x] [`arrow-row/src/variable.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-row/src/variable.rs): [`decode_binary`](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-row/src/variable.rs#L287) builds offsets with `BufferBuilder::` (done in #10851)
- [ ] [`arrow-select/src/zip.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-select/src/zip.rs): [byte-array zip path](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-select/src/zip.rs#L562) uses `OffsetBufferBuilder::` (PR open: #10913)
- [x] ~[`arrow-avro/src/reader/record.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-avro/src/reader/record.rs): [`Decoder` enum variants](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-avro/src/reader/record.rs#L245) (`Binary`/`String`/`StringView`/`Array`/`Map`) hold `OffsetBufferBuilder` as long-lived decoder state (~15 sites; a larger refactor than the others since the builders are not per-call scratch buffers)~ won't do: per discussion in #10923, keeping `OffsetBufferBuilder` here is clearer and the usages are not in a tight hot loop
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start with the unchecked callsites in arrow-arith/src/arity.rs and arrow-select/src/zip.rs, especially try_binary and the byte-array zip path. Compare the completed conversions listed in the issue, then verify that each targeted builder usage is replaced where appropriate and the relevant crate tests pass; the zip work already has PR #10913 open.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100