Enable automatic memory accounting using the memory pool when creating arrays
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When tracking memory usage with arrow memory pools (introduced in https://github.com/apache/arrow-rs/pull/7303, tracked by EPIC https://github.com/apache/arrow-rs/issues/8137), users must manually call `claim()` on buffers after array creation. It could be easy to forget to call `claim()` on newly created arrays, leading to potentially under-reporting memory usage, also it requires having direct access to `ArrayData` just to access buffers for claiming, which in some cases is complicated.
It would be much more ergonomic and reliable to automatically claim buffers at array creation time.
**Describe the solution you'd like**
Add optional `MemoryPool` parameters to array constructors and builders, so buffers are automatically claimed at creation time. For example we could have: `PrimitiveArray:: try_new_with_pool()` in array costructors, `PrimitiveBuilder::new_with_pool` that claim buffers once we call `finish()` etc.
**Additional context**
This feature is needed to improve memory accounting and tracking in downstream projects like DataFusion. Related discussions:
- https://github.com/apache/datafusion/issues/16841
Contributor guide
Research direction
Start by surveying the Rust array constructors and builders described in the issue, along with the existing MemoryPool and buffer claim APIs. Define the scope of optional pool-aware construction and builder finishing, then verify that newly created arrays claim their buffers without manual ArrayData access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100