apache / apache/arrow-rs

Parquet metadata as flatbuffers

Open
#9,041 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Reading wide tables spends relatively much time on decoding thrift metadata. There is an ongoing discussion and an active proposal in the Parquet community to address this by storing footer metadata as flatbuffers. See [past mailing list discussions](https://lists.apache.org/list?dev@parquet.apache.org:2025-9:[DISCUSS]%20flatbuf%20footer), parquet-format [community discussion](https://github.com/apache/parquet-format/issues/530), active PR by @alkis and @Jiayi-Wang-db -> https://github.com/apache/arrow/pull/48431 and Alkis' [design doc](https://docs.google.com/document/d/1kZS_DM_J8n6NKff3vDQPD1Y4xyDdRceYFANUE0bOfb0/edit?usp=sharing).

**Describe the solution you'd like**

We would want to test an implementation Rust (and C++), to show performance and interoperability as well as have two reference implementations before discussing this further in the Parquet community.

Contributor guide

Open the contributing guide

Research direction

No repository files or tests are named. Start by reading the linked design document, Parquet community discussion, and referenced Arrow pull request, then inspect the existing Parquet metadata handling. Done means a Rust implementation with performance and interoperability evidence, alongside the proposed C++ reference implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.