apache / apache/hudi

Rollback may omit invalid files when commitMetadata is not completed for MOR

Open
#15,883 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Now ,Use listingBasedRollbackStrategy may omit invalid files when commitMetadata is not completed.The reason for this problem is due to use instantToRollback timestamp and the baseCommitTime of the logFile to judge whether the Logfiles is valid.

{code:java}
// commit is instant time which should be rollback
// in most cases BaseCommitTime may not equals commit
(path) -> {
if (path.toString().endsWith(basefileExtension)) {
String fileCommitTime = FSUtils.getCommitTime(path.getName());
return commit.equals(fileCommitTime);
} else if (FSUtils.isLogFile(path)) {
// Since the baseCommitTime is the only commit for new log files, it's okay here
String fileCommitTime = FSUtils.getBaseCommitTimeFromLogPath(path);
return commit.equals(fileCommitTime);
}
return false;
};
{code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6029
- Type: Bug

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the implementation of listingBasedRollbackStrategy and read how it determines valid base and log files during rollback. Reproduce the MOR case where commitMetadata is incomplete, then verify that rollback includes the invalid files omitted by the current instantToRollback and baseCommitTime checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.