[SUPPORT] Unify schema evolution handling for base and log files in HoodieLsmFileGroupReader
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
### Problem
`HoodieLsmFileGroupReader` currently handles schema evolution for base files and native log files through separate paths. This introduces duplicated schema resolution and projection steps, makes the reader flow harder to maintain, and can cause base and log sorted runs to behave differently for the same schema change.
Parent issue: #14310
### Proposed work
- Define one schema-evolution flow for base-file and native-log sorted runs in `HoodieLsmFileGroupReader`.
- Reuse the common schema handling and evolution transformer logic used by `HoodieFileGroupReader` where applicable.
- Avoid redundant projection stages between physical, file, required, and requested schemas.
- Avoid reading native-log footer schemas when schema evolution is disabled.
- Keep projection, predicate pushdown, delete handling, merge ordering, and key-field requirements consistent across base and native-log inputs.
### Acceptance criteria
- Base-only, log-only, and base-plus-native-log file groups share a consistent schema-evolution contract.
- Schema evolution enabled and disabled paths are covered by tests.
- Tests cover field addition/removal/reordering and compatible type evolution for both base and native-log runs.
- No extra native-log footer schema read is performed when schema evolution is disabled.
- Existing legacy inline-log fallback behavior is unchanged.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading HoodieLsmFileGroupReader and compare its base-file and native-log schema handling with the common logic in HoodieFileGroupReader. Locate the existing tests for these readers and run them before changing behavior. Done means consistent schema evolution and projection across base-only, log-only, and combined inputs, with the listed evolution cases covered and no footer schema read when evolution is disabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100