apache / apache/parquet-java

Add rewriter version to Parquet footer

Open
#2,649 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Major Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

Parquet footer records the writer's version in the field of 'create-by'. As we introduce several rewrites, the new file is written partially by the rewriter. In this case, we need to record the rewriter's version also.

Some questions (about a common rewriter) we need to answer before step forward:

What would be the place of the rewriter versions? (New specific field or key-value metadata? Which key shall we use?)
Shall we somehow also save what the rewriter has done? How?
At what level shall we copy the original created_by field and what level shall we write the version of the rewriter to that field instead? (What different levels are possible?)
From the introduction of this rewriter(s) field in case of any related writer version dependent fix we need to check this field as well and not only the created_by one.

**Reporter**: [Xinli Shang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=shangx@uber.com) / @shangxinli
**Assignee**: [Xinli Shang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=shangx@uber.com) / @shangxinli

**Note**: *This issue was originally created as [PARQUET-2093](https://issues.apache.org/jira/browse/PARQUET-2093). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no source files, tests, or entry points. Start by examining Parquet footer handling and the existing created_by field, then resolve where rewriter version and rewrite details belong, how created_by should be preserved, and how version-dependent fixes should detect them. Done means these design questions are decided and the resulting footer behavior is implemented and tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.