[C++][Parquet] Do not duplicate Schema metadata in Parquet schema metadata and serialized ARROW:schema value
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Metadata values are being duplicated in the Parquet file footer — we should either only store them in ARROW:schema or the Parquet schema metadata. Removing them from the Parquet schema metadata may break applications that are expecting that metadata to be there when serialized from Arrow, so dropping the keys from ARROW:schema is probably a safer choice
**Reporter**: [Wes McKinney](https://issues.apache.org/jira/browse/ARROW-14303) / @wesm
**Note**: *This issue was originally created as [ARROW-14303](https://issues.apache.org/jira/browse/ARROW-14303). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Review the Parquet schema metadata and serialized ARROW:schema handling described in the issue, using the migration documentation in issue #14542 for compatibility context. Identify which metadata keys are duplicated and define completion as storing them in one location without breaking existing consumers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100