apache / apache/hudi

Fix rollbacks with MDT for MOR data table with log files

Open
#16,208 1 comment 0 reactions 1 assignee Assigned to @nsivabalan View on GitHub
area:metadata-table from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

There are few rollback scenarios, where some log files from data table could be missed to sync to MDT. Esply for cleaner purpose, every valid file from data table (which could be seen with fs.listStatus), should be synced to MDT. we can't afford to miss any log files. 

 

Two major gaps which needs to be fixed. 

1. log files from original commit being rolled back. 

for eg, t5.dc fails mid-way in DT which added lf2. 

we start a rollback commit t6.rb. when t6 syncs to MDT, we should also track lf2 and ensure we sync to MDT. 

2. log files added by previous attempts of rollbacks. 

in the above scenario, rollback could have added a log file (rollback command block) called lf3. 

but if the rollback failed and is re-attempted, it could add another file called lf4. So, when this rollback syncs to MDT, we need to somehow ensure lf3 is also synced w/o a miss. 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6761
- Type: Bug
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.