HoodieFileGroupReader should already know the schema that it is reading
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
The current implementation in HudiPageSourceProvider requires us to pass a requestedSchema into HoodieFileGroupReader. In theory, all schema (evolution) related handling should be handled by HoodieFileGroupReader and all HudiPageSourceProvider needs to do is pass the required columns that needs to be read (for column pruning/projection)
A followup cleanup within HoodieFileGroupReader is required to expose this interface and implement this functionality.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9429
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-2687
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in HudiPageSourceProvider and HoodieFileGroupReader to trace how requestedSchema is passed today and where column pruning/projection is determined. Read the linked HUDI-9429 task for context, then define the reader interface around required columns while keeping schema-evolution handling inside HoodieFileGroupReader. Done means the provider no longer supplies the requested schema and the reader still reads the required columns correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100