Support reporting heap size of custom `PageIndexProvider`
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- Follow on to https://github.com/apache/arrow-rs/pull/10842 from @etseidl
We added the [`PageIndexProvider`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/page_index.rs#L139) trait so users can supply custom page index representations. However there is no way for `PageIndexProvider` to report their memory usage in [`ParquetMetaData::memory_size`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/mod.rs#L231)
Since one of the main motivations for the provider trait is more efficient caching of parquet metadata, callers need accurate memory accounting for custom providers as well.
See here for some prior attempts at exposing `HeapSize` publicly:
- https://github.com/apache/arrow-rs/pull/9138
- https://github.com/apache/arrow-rs/pull/9842
**Describe the solution you'd like**
Add a way for the `PageIndexProvider` trait to report its heap size so `ParquetMetaData::memory_size` can include it. Ideally implementors could re-use the [`HeapSize`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/memory.rs#L36) machinery the parquet crate already implements.
**Describe alternatives you've considered**
- Add a required (or defaulted-to-zero) `heap_size(&self) -> usize` method directly on `PageIndexProvider`, without exposing the `HeapSize` trait publicly.
- Make the `HeapSize` trait public (per the prior attempts above) and require `PageIndexProvider: HeapSize`.
**Additional context**
- Original review discussion: https://github.com/apache/arrow-rs/pull/10842#discussion_r3913858985
Contributor guide
Research direction
Read parquet/src/file/metadata/page_index.rs, parquet/src/file/metadata/mod.rs, and parquet/src/file/metadata/memory.rs, starting with PageIndexProvider, ParquetMetaData::memory_size, and HeapSize. Review the linked prior attempts and original discussion before choosing the public API. Done means custom page-index memory is included in ParquetMetaData::memory_size without breaking existing providers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100