Re-think restore for MOR table
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
As of now, our restore is triggering N rollbacks in reverse chronological order.
for MOR table, we need to re-think this, since our rollbacks are log appends, even if we want to restore to a delta commit just before last compaction, instead of directly cleaning up the lastest file slice, we are triggering rollbacks of N delta commits and finally cleaning up the entire file slice when the compaction instant is being rolledback.
we can optimize things better here for MOR table. we just need to re-think restore from scratch w/o the need for going thru N rollbacks.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5722
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing restore for MOR tables, including the rollback sequence for delta commits and the compaction instant that cleans up the file slice. Done means the restore design avoids iterating through N rollbacks while still restoring correctly to a delta commit before the last compaction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100