apache / apache/datafusion

Populate FileWriteMetadata.format_metadata with Thrift-serialized Parquet footer for FFI consumers

Open
#24,010 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Background

PR #23656 added `FileWriteMetadata` with a `format_metadata: Option` field designed to carry format-specific metadata (e.g., Thrift-serialized Parquet `FileMetaData`). The current implementation sets this to `None` because the typed `ParquetMetaData` is already accessible via `ParquetSink::written()` for Rust consumers.

## Proposal

When a concrete FFI use case emerges (e.g., Python via PyO3, or C via `datafusion-ffi`), serialize the Parquet `FileMetaData` using Thrift Compact Protocol and populate `format_metadata` so that non-Rust consumers can access per-file column statistics without re-reading file footers.

## Context

- PR: #23656
- Review comment: https://github.com/apache/datafusion/pull/23656#discussion_r3683548423

Contributor guide

Open the contributing guide

Research direction

Start by reviewing PR #23656 and the FileWriteMetadata and ParquetSink::written() references described here. Confirm a concrete FFI consumer and agree on the serialized metadata contract; done means the Thrift Compact Protocol footer is populated for that use case and non-Rust consumers can access the per-file statistics without rereading footers.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.