apache / apache/arrow-rs

Enable automatic memory accounting using the memory pool when creating arrays

Open
#8,938 24 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

When tracking memory usage with arrow memory pools (introduced in https://github.com/apache/arrow-rs/pull/7303, tracked by EPIC https://github.com/apache/arrow-rs/issues/8137), users must manually call `claim()` on buffers after array creation. It could be easy to forget to call `claim()` on newly created arrays, leading to potentially under-reporting memory usage, also it requires having direct access to `ArrayData` just to access buffers for claiming, which in some cases is complicated.

It would be much more ergonomic and reliable to automatically claim buffers at array creation time.

**Describe the solution you'd like**

Add optional `MemoryPool` parameters to array constructors and builders, so buffers are automatically claimed at creation time. For example we could have: `PrimitiveArray:: try_new_with_pool()` in array costructors, `PrimitiveBuilder::new_with_pool` that claim buffers once we call `finish()` etc.

**Additional context**

This feature is needed to improve memory accounting and tracking in downstream projects like DataFusion. Related discussions:
- https://github.com/apache/datafusion/issues/16841

Contributor guide

Open the contributing guide

Research direction

Start by surveying the Rust array constructors and builders described in the issue, along with the existing MemoryPool and buffer claim APIs. Define the scope of optional pool-aware construction and builder finishing, then verify that newly created arrays claim their buffers without manual ArrayData access.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.