apache / apache/arrow

[C++][Parquet] Do not duplicate Schema metadata in Parquet schema metadata and serialized ARROW:schema value

Open
#29,877 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Metadata values are being duplicated in the Parquet file footer — we should either only store them in ARROW:schema or the Parquet schema metadata. Removing them from the Parquet schema metadata may break applications that are expecting that metadata to be there when serialized from Arrow, so dropping the keys from ARROW:schema is probably a safer choice

**Reporter**: [Wes McKinney](https://issues.apache.org/jira/browse/ARROW-14303) / @wesm

**Note**: *This issue was originally created as [ARROW-14303](https://issues.apache.org/jira/browse/ARROW-14303). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Review the Parquet schema metadata and serialized ARROW:schema handling described in the issue, using the migration documentation in issue #14542 for compatibility context. Identify which metadata keys are duplicated and define completion as storing them in one location without breaking existing consumers.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.