apache / apache/arrow-rs

Support reporting heap size of custom `PageIndexProvider`

Open
#11,028 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

- Follow on to https://github.com/apache/arrow-rs/pull/10842 from @etseidl

We added the [`PageIndexProvider`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/page_index.rs#L139) trait so users can supply custom page index representations. However there is no way for `PageIndexProvider` to report their memory usage in [`ParquetMetaData::memory_size`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/mod.rs#L231)

Since one of the main motivations for the provider trait is more efficient caching of parquet metadata, callers need accurate memory accounting for custom providers as well.

See here for some prior attempts at exposing `HeapSize` publicly:
- https://github.com/apache/arrow-rs/pull/9138
- https://github.com/apache/arrow-rs/pull/9842

**Describe the solution you'd like**

Add a way for the `PageIndexProvider` trait to report its heap size so `ParquetMetaData::memory_size` can include it. Ideally implementors could re-use the [`HeapSize`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/memory.rs#L36) machinery the parquet crate already implements.

**Describe alternatives you've considered**

- Add a required (or defaulted-to-zero) `heap_size(&self) -> usize` method directly on `PageIndexProvider`, without exposing the `HeapSize` trait publicly.
- Make the `HeapSize` trait public (per the prior attempts above) and require `PageIndexProvider: HeapSize`.

**Additional context**

- Original review discussion: https://github.com/apache/arrow-rs/pull/10842#discussion_r3913858985

Contributor guide

Open the contributing guide

Research direction

Read parquet/src/file/metadata/page_index.rs, parquet/src/file/metadata/mod.rs, and parquet/src/file/metadata/memory.rs, starting with PageIndexProvider, ParquetMetaData::memory_size, and HeapSize. Review the linked prior attempts and original discussion before choosing the public API. Done means custom page-index memory is included in ParquetMetaData::memory_size without breaking existing providers.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.