apache / apache/parquet-format
Design a general extension mechanism for Parquet
- Dominant language
- Thrift
- Stars
- 2.6k
- Forks
- 508
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 4
Description
Design a general extension mechanism for Parquet with the following requirements:
- old readers must be able to parse files written with the extension
- new readers must be able to parse files written with/without the extension and deterministically detect if the extension is present or not
Initial ideas: https://docs.google.com/document/d/1PQpY418LkIDHMFYCY8ne_G-CFpThK15LLpzWYbc7rFU/edit
**Reporter**: [Alkis Evlogimenos](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=alkis) / @alkis
#### PRs and other links:
- [GitHub Pull Request #254](https://github.com/apache/parquet-format/pull/254)
**Note**: *This issue was originally created as [PARQUET-2492](https://issues.apache.org/jira/browse/PARQUET-2492). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
Research direction
Start with the linked design document and Pull Request #254, then compare the proposal with the issue's compatibility requirements. Done means an agreed general extension mechanism that lets old readers parse extended files and lets new readers deterministically detect whether the extension is present.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100