apache / apache/parquet-format
Add FlatBuffers footer to speed up metadata parsing
- Dominant language
- Thrift
- Stars
- 2.6k
- Forks
- 508
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 4
Description
### Describe the enhancement requested
@alkis is spearheading the improvement of Parquet footer metadata, with a proposal for adding an optional FlatBuffers based footer to the Parquet format. Andrew Bell noted on the mailing list there was no issue to track this, so filing one here
Quoting from the [Proposal Document](https://docs.google.com/document/d/1kZS_DM_J8n6NKff3vDQPD1Y4xyDdRceYFANUE0bOfb0/edit?tab=t.0)
> Tabular data stored in Parquet files with thousands of leaf columns are becoming increasingly common, fueled by AI datasets. Decoding Parquet files always involves a two step process. First the Parquet metadata (aka footer) is fetched and resolved and then the row groups/column chunks are fetched and decoded. The two steps are serial because of the natural data dependency between them. Thus improving the time it takes to resolve Parquet metadata will allow engines to be more efficient to process Parquet files.
Related mailing list discussion: https://lists.apache.org/thread/j9qv5vyg0r4jk6tbm6sqthltly4oztd3
Contributor guide
Research direction
Start with the linked Proposal Document and the related mailing-list discussion to understand the proposed optional footer and its format implications. The issue does not name implementation files or tests; a useful next step is to identify the affected Parquet format definitions and establish the scope and acceptance criteria for adding the footer.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100