Fix deltastreamer multi-writer checkpoint mechanism
Open
area:ingest
from-jira
priority:blocker
type:bug
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
if we have multiple deltastreamer writers w/ out of order commits, chances that checkpoint management can miss things and can cause dups. we need to fix it properly.
we need to make a fix similar to how we fixed it for spark streaming sink https://github.com/apache/hudi/pull/8558
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6159
- Type: Bug
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.