Introduce an alignment-guaranteeing Vec wrapper for use in buffer-building optimizations
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
### Is your feature request related to a problem or challenge?
Several performance optimizations in arrow-rs (e.g. #10812, take on List) would naturally reach for Vec or Vec, Rust's stdlib Vec is heavily optimized by the compiler and benefits from inlining, LLVM vectorization hints, and well-understood growth patterns. However, any buffer that will be reinterpreted as a typed `ScalarBuffer` must satisfy T's alignment requirement. `Vec` only guarantees 1-byte alignment, causing panics like Memory pointer is not aligned with the specified scalar type
### Describe the solution you'd like
Introduce a thin newtype in arrow-buffer along the lines of:
```
#[repr(align(64))]
struct Align64([u8; 64]);
pub struct AlignedVec(Vec);
```
that exposes a Vec-compatible interface (push, extend_from_slice, len, as_slice, etc.) while guaranteeing 64-byte alignment. This would let optimization hot paths use idiomatic Vec patterns and get full compiler/LLVM optimization benefits without sacrificing Arrow's alignment invariants.
*this would surely need more work than this, but just a loose idea*
### Describe alternatives you've considered
sticking with MutableBuffer
### Additional context
this has been a common issue I've encountered when trying to optimize portions of the code base. #10812 recently occurred where switching from `Vec` to MutableBuffer caused a [30% regression](https://github.com/apache/arrow-rs/pull/10812#issuecomment-5429213753)
Contributor guide
Research direction
Start in arrow-buffer by reviewing MutableBuffer, ScalarBuffer, and the alignment assumptions behind reinterpreting buffers. Compare the optimization context in #10812 and the List take path, then determine the smallest Vec-compatible API that guarantees alignment; done means the wrapper preserves Arrow's alignment invariants and supports the intended hot paths without the reported regression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100