"end to end" test for reading parquet using custom PageIndexProvider
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
https://github.com/apache/arrow-rs/pull/10842 introduces the [`PageIndexProvider`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/page_index.rs) trait so that applications can plug in custom page index implementations (for example, sparse or lazily-loaded indexes) into [`ParquetMetaData`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/mod.rs).
However, no test currently drives reading with a custom [`PageIndexProvider`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/page_index.rs) through the actual read path.
Several places in the code use `as_any().downcast_ref::()` and take a different path for custom providers (for example [`ParquetMetaData::memory_size`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/mod.rs), `PartialEq for ParquetMetaData`, and [`ParquetMetaDataWriter`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/writer.rs)), so adding specific reader coverage would be good.
- The [`custom_page_index.rs`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/examples/custom_page_index.rs) example implements `OnDemandPageIndexProvider`, but it only prints index availability, and examples are compiled — not executed — in CI.
- The existing reader tests exercise the built-in `PageIndex` implementation only.
**Describe the solution you'd like**
Add an integration test that:
1. Writes a parquet file with page indexes (multiple row groups, multiple columns, multiple pages per column).
2. Loads the metadata with [`PageIndexPolicy::Skip`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/reader.rs), then installs a custom `PageIndexProvider` implementation (similar to the example's [`OnDemandPageIndexProvider`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/examples/custom_page_index.rs)) via [`ParquetMetaDataBuilder::set_page_index`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/mod.rs), populated for only a subset of row groups / columns.
3. Reads data through [`ParquetRecordBatchReader`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/arrow/arrow_reader/mod.rs) (and/or the async reader / push decoder) with a [`RowSelection`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/arrow/arrow_reader/selection.rs) that triggers page skipping, and verifies:
- correct results are returned,
- the offset-index-driven fetch path is used for columns where the provider returns `Some`,
- columns where the provider returns `None` fall back to whole-column-chunk fetching.
**Describe alternatives you've considered**
Converting the [`custom_page_index.rs`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/examples/custom_page_index.rs) example into an executed test, or adding assertions to it and running it in CI.
**Additional context**
- Follow-up from review of https://github.com/apache/arrow-rs/pull/10842
Contributor guide
Assessment
This issue has not been assessed yet.