apache / apache/hudi

Re-think restore for MOR table

Open
#15,766 0 comments 0 reactions 0 assignees View on GitHub
area:dev-experience area:table-service area:writer component:rollback from-jira priority:critical type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

As of now, our restore is triggering N rollbacks in reverse chronological order. 

for MOR table, we need to re-think this, since our rollbacks are log appends, even if we want to restore to a delta commit just before last compaction, instead of directly cleaning up the lastest file slice, we are triggering rollbacks of N delta commits and finally cleaning up the entire file slice when the compaction instant is being rolledback. 

 

we can optimize things better here for MOR table. we just need to re-think restore from scratch w/o the need for going thru N rollbacks. 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5722
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing restore for MOR tables, including the rollback sequence for delta commits and the compaction instant that cleans up the file slice. Done means the restore design avoids iterating through N rollbacks while still restoring correctly to a delta commit before the last compaction.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.