Auto repair tool for MDT out of sync
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Can we write a spark-submit to repair any out of sync issues w/ MDT. for eg, if MDT validation failed for a given table, we don't have a good way to fix the MDT.
So, we should develop a sparksubmit job which will try to deduce from which commit the out of sync happens and try to fix just the delta.
idea here is:
Try running validation job for latest files at every commit starting from latest in reverse chronological order. At some point validation will succeed. Lets call it commit N.
we can add savepoint to MDT at commit N and restore the table to that commit N.
and then we can take any new commits after commitN from data table and apply them one by one to MDT.
Once complete, we can run validation tool again to ensure its in good shape.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5436
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-1292
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.