apache / apache/arrow-rs

Add metadata index for Parquet files

Open
#8,713 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Related to #5855 and #5853.

One of the pain points of reading Parquet files is the all-or-nothing nature of the file metadata, which is stored as a Thrift encoded blob in the file's footer. A traditional parser built from Thrift generated code will decode the entire `FileMetaData` structure, which can be very costly with extremely large schemas. The new parsing code introduced recently (#5854) can reduce this cost some by skipping unwanted structures, but as currently implemented it still needs to process the Thrift framing even if not fully decoding everything.

**Describe the solution you'd like**
One solution to the above is to provide an index into the serialized metadata so that only the structures requested are parsed. A full implementation of this would be used along with either row group selections or column projections, and would also be of use for predicate processing (only read column chunk statistics for columns present in the predicate, for instance). This will also need the options object detailed in #8643.

Such an index could be embedded in the `FileMetaData` in the manner described in the [Binary Protocol Extensions](https://github.com/apache/parquet-format/blob/master/BinaryProtocolExtensions.md) section of the Parquet specification.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the new parsing code from #5854, the options object described in #8643, and the Parquet Binary Protocol Extensions specification. Define how an index embedded in FileMetaData would support selective parsing for row groups, projected columns, and predicate statistics; done should include a tested design and implementation that avoids unnecessary metadata processing.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.