apache / apache/arrow

[Parquet][C++] PageIndex is useless with current API

Open
#45,284 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

The `ParquetFileReader` provides a `PageIndexReader` via which we can eventually get to a `ColumnIndex` and an `OffsetIndex` - so far so good. Those indexes provide page based information, but in virtually all APIs the concept of pages is completely abstracted away. For higher level APIs that makes sense, but even if we go down to the level of the `PageReader` we can only read all pages serially one after the other. The only way I found to skip some pages is via the `PageReader`'s data page filter, but that only operates on the page's metadata and does not utilize the index. I did not find a way to load a specific page (e.g.,via index or file offset). But then I don't see how one can utilize the PageIndex with the current API. Did I miss anything?

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.