Populate FileWriteMetadata.format_metadata with Thrift-serialized Parquet footer for FFI consumers
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
## Background
PR #23656 added `FileWriteMetadata` with a `format_metadata: Option` field designed to carry format-specific metadata (e.g., Thrift-serialized Parquet `FileMetaData`). The current implementation sets this to `None` because the typed `ParquetMetaData` is already accessible via `ParquetSink::written()` for Rust consumers.
## Proposal
When a concrete FFI use case emerges (e.g., Python via PyO3, or C via `datafusion-ffi`), serialize the Parquet `FileMetaData` using Thrift Compact Protocol and populate `format_metadata` so that non-Rust consumers can access per-file column statistics without re-reading file footers.
## Context
- PR: #23656
- Review comment: https://github.com/apache/datafusion/pull/23656#discussion_r3683548423
Contributor guide
Research direction
Start by reviewing PR #23656 and the FileWriteMetadata and ParquetSink::written() references described here. Confirm a concrete FFI consumer and agree on the serialized metadata contract; done means the Thrift Compact Protocol footer is populated for that use case and non-Rust consumers can access the per-file statistics without rereading footers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100