Not able to do backwards compatible projection
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
Currently the following scenarios fail while using projections. This makes it hard to do backwards compatible schema changes like adding an optional field or making a required field optional.
1. Adding an optional field to an existing thrift schema and making it part of the projection for reading data written using older thrift.
2. Projecting a required field using optional.
**Reporter**: [Mouli Mukherjee](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=moulim)
#### PRs and other links:
- [GitHub Pull Request #558](https://github.com/apache/parquet-mr/pull/558)
**Note**: *This issue was originally created as [PARQUET-1464](https://issues.apache.org/jira/browse/PARQUET-1464). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Review the projection scenarios described in the issue, especially adding an optional Thrift field and projecting a required field as optional. Read GitHub Pull Request #558 and the linked migration documentation first, then use those materials to determine the affected projection behavior and what compatibility tests are still missing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100