Completion Markers for handling spark retries
Open
from-jira
priority:high
status:pr-available
type:bug
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
During spark stage retries, spark driver may have all the information to reconcile the commit and proceed with next steps, while a stray executor may still be writing to a data file and complete later (before the JVM exit).
Extra files left on the dataset, excluded from reconcile commit step could show up as data quality issue for query engines with duplicate records.
This change brings completion markers which tries to prevent the dataset from experiencing data quality issues, in such corner case scenarios.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6416
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-7967
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.