Improve Parquet metadata reads by avoiding fixed-size prefetch
- Dominant language
- Rust
- Stars
- 197
- Forks
- 92
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 91
Description
### Problem
The async Parquet reader currently uses a fixed 512 KiB metadata prefetch hint. For HDFS through OpenDAL’s native service, this can read substantially more data than the footer metadata requires.
Internal HDFS testing on a representative workload showed better read performance when following parquet-hadoop’s seek-based sequence instead: read the 8-byte footer, then fetch the exact footer metadata and any requested page-index range.
### Proposed change
- Keep the existing 512 KiB metadata prefetch for object stores and unknown or custom backends.
- Use exact metadata range reads only for backends where positioned reads are known to be cheap, including HDFS native, local filesystem, and memory.
- Keep page-index loading demand-driven.
- Avoid loading `OffsetIndex` for an empty external row selection.
- Keep the backend classification and Parquet metadata-read policy crate-private instead of expanding the public `FileRead` contract.
- Add range-tracking tests for exact footer, metadata, and `OffsetIndex` reads, and for retained prefetch behavior.
### Validation
- Internal HDFS comparison on the target workload showed an improvement with exact metadata reads.
- Automated tests verify the expected request ranges and preserve the conservative object-store behavior.
- A standalone reproducible benchmark report is not required for this issue; additional performance measurements can be documented separately if needed.
Implementation: #686
Contributor guide
Research direction
Start with implementation issue #686 and trace the async Parquet reader’s metadata-read path. Inspect the crate-private backend classification, metadata-read policy, and existing FileRead contract before adding range-tracking tests. Done means exact footer, metadata, and demand-driven OffsetIndex ranges for eligible backends while retained prefetch behavior is verified for object stores and unknown backends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100