Rollback may omit invalid files when commitMetadata is not completed for MOR
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Now ,Use listingBasedRollbackStrategy may omit invalid files when commitMetadata is not completed.The reason for this problem is due to use instantToRollback timestamp and the baseCommitTime of the logFile to judge whether the Logfiles is valid.
{code:java}
// commit is instant time which should be rollback
// in most cases BaseCommitTime may not equals commit
(path) -> {
if (path.toString().endsWith(basefileExtension)) {
String fileCommitTime = FSUtils.getCommitTime(path.getName());
return commit.equals(fileCommitTime);
} else if (FSUtils.isLogFile(path)) {
// Since the baseCommitTime is the only commit for new log files, it's okay here
String fileCommitTime = FSUtils.getBaseCommitTimeFromLogPath(path);
return commit.equals(fileCommitTime);
}
return false;
};
{code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6029
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the implementation of listingBasedRollbackStrategy and read how it determines valid base and log files during rollback. Reproduce the MOR case where commitMetadata is incomplete, then verify that rollback includes the invalid files omitted by the current instantToRollback and baseCommitTime checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100