apache / apache/arrow-rs

Add a feature to allow creating Buffers from Vec with custom allocators (allocator_api)

Open
#3,960 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

in #3920 and #3917 support was added to to create buffers from standard Rust vectors. The currently instable `allocator_api` feature extends `Vec` to support custom allocators, using functions such as [`new_in`](https://doc.rust-lang.org/std/vec/struct.Vec.html#method.new_in). In our product we are using such custom allocators to track the memory usage of individual queries. I'd like to add a similarly named feature to arrow-rs which would generalize the `Buffer::from_vec` and `MutableBuffer::from_vec` functions.

**Describe the solution you'd like**

- Add a feature named `allocator-api` to `arrow-buffer`
- If that feature is active, a different impl of `from_vec` would be enabled (via cfg attribute), which has a generic parameter for the allocator
- `Deallocation::Standard` would additionally store the allocator (note the default `Global` allocator is zero-sized)
- The feature would require a nightly rust compiler, just as the feature of the standard library

**Describe alternatives you've considered**

Something similar can be achieved using `Buffer::from_custom_allocation` but requires unsafe and dealing with pointers.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start in the arrow-buffer crate by locating Buffer::from_vec, MutableBuffer::from_vec, and Deallocation::Standard. Review the existing standard-vector implementation and the allocator_api requirements, then check how the feature is configured and how nightly compilation is handled. Done means custom-allocator vectors are supported through the requested feature without regressing standard allocation behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.