apache / apache/paimon-rust

Improve Parquet metadata reads by avoiding fixed-size prefetch

Open
#687 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
197
Forks
92
Avg merge
1d 16h
Merged PRs (30d)
91

Description

### Problem

The async Parquet reader currently uses a fixed 512 KiB metadata prefetch hint. For HDFS through OpenDAL’s native service, this can read substantially more data than the footer metadata requires.

Internal HDFS testing on a representative workload showed better read performance when following parquet-hadoop’s seek-based sequence instead: read the 8-byte footer, then fetch the exact footer metadata and any requested page-index range.

### Proposed change

- Keep the existing 512 KiB metadata prefetch for object stores and unknown or custom backends.
- Use exact metadata range reads only for backends where positioned reads are known to be cheap, including HDFS native, local filesystem, and memory.
- Keep page-index loading demand-driven.
- Avoid loading `OffsetIndex` for an empty external row selection.
- Keep the backend classification and Parquet metadata-read policy crate-private instead of expanding the public `FileRead` contract.
- Add range-tracking tests for exact footer, metadata, and `OffsetIndex` reads, and for retained prefetch behavior.

### Validation

- Internal HDFS comparison on the target workload showed an improvement with exact metadata reads.
- Automated tests verify the expected request ranges and preserve the conservative object-store behavior.
- A standalone reproducible benchmark report is not required for this issue; additional performance measurements can be documented separately if needed.

Implementation: #686

Contributor guide

Open the contributing guide

Research direction

Start with implementation issue #686 and trace the async Parquet reader’s metadata-read path. Inspect the crate-private backend classification, metadata-read policy, and existing FileRead contract before adding range-tracking tests. Done means exact footer, metadata, and demand-driven OffsetIndex ranges for eligible backends while retained prefetch behavior is verified for object stores and unknown backends.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.