Add a feature to allow creating Buffers from Vec with custom allocators (allocator_api)
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
in #3920 and #3917 support was added to to create buffers from standard Rust vectors. The currently instable `allocator_api` feature extends `Vec` to support custom allocators, using functions such as [`new_in`](https://doc.rust-lang.org/std/vec/struct.Vec.html#method.new_in). In our product we are using such custom allocators to track the memory usage of individual queries. I'd like to add a similarly named feature to arrow-rs which would generalize the `Buffer::from_vec` and `MutableBuffer::from_vec` functions.
**Describe the solution you'd like**
- Add a feature named `allocator-api` to `arrow-buffer`
- If that feature is active, a different impl of `from_vec` would be enabled (via cfg attribute), which has a generic parameter for the allocator
- `Deallocation::Standard` would additionally store the allocator (note the default `Global` allocator is zero-sized)
- The feature would require a nightly rust compiler, just as the feature of the standard library
**Describe alternatives you've considered**
Something similar can be achieved using `Buffer::from_custom_allocation` but requires unsafe and dealing with pointers.
**Additional context**
Contributor guide
Research direction
Start in the arrow-buffer crate by locating Buffer::from_vec, MutableBuffer::from_vec, and Deallocation::Standard. Review the existing standard-vector implementation and the allocator_api requirements, then check how the feature is configured and how nightly compilation is handled. Done means custom-allocator vectors are supported through the requested feature without regressing standard allocation behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100