Support incremental & streaming query for pre-8 (LAYOUT_VERSION_1) tables by treating instant time as completion time in the 1.x reader
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Feature Description
For a LAYOUT_VERSION_1 timeline (table versions 5–7, 0.12–0.16), an instant's completion time is, backward-compatibly, its instant (request) time — a V1 timeline simply does not persist a separate completion time. The 1.x incremental read stack (IncrementalQueryAnalyzer → CompletionTimeQueryView) does not yet apply this equivalence: CompletionTimeQueryViewV1.getInstantTimes throws "not yet implemented", so incremental/streaming reads against these tables fail.
This umbrella makes the 1.x reader query these tables by reusing the existing completion-time range logic with "completion time = instant time" for V1, rather than adding a parallel V1-specific implementation. Sub-tasks: the common query view (the equivalence), and two version-specific read-path bugs on the Spark and Flink sides.
sub tasks:
PR1:feat(common): make CompletionTimeQueryView query LAYOUT_VERSION_1 timelines by using instant time as completion time
PR2:fix(spark): propagate instant range to the file-group reader on the MOR incremental read path
PR3:fix(flink): use correct log-reader high watermark for pre-8 MOR incremental read
### User Experience
**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples
### Hudi RFC Requirements
**RFC PR link:** (if applicable)
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with IncrementalQueryAnalyzer and CompletionTimeQueryView, then inspect CompletionTimeQueryViewV1.getInstantTimes and the Spark and Flink MOR incremental read paths named in the subtasks. Done means pre-8 LAYOUT_VERSION_1 tables support incremental and streaming reads through the common completion-time logic and both version-specific read-path fixes are covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100