Flink Checkpoint -1 is written to hudi metadata
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
When Hudi Flink StreamWriteOperatorCoodinator try to recommit the last in flight instant if the write metadata checkpoint successfully but was not committed due to some rare cases, the checkpoint id is set to -1 and committed to metadata
**What you expected:**
When StreamWriteOperatorCoodinator try to commit an instance with checkpoint = -1, we should look up the last commit and find the corresponding checkpoint Id A inside and commit the instant with checkpoint A +1.
**Steps to reproduce:**
1. Flink hudi ingestion job successfully finished checkpoint
2. Hudi Fail due to storage issue
3. Job restart and try to recommit
### Environment
**Hudi version:**
0.14 to main. The behavior is the same.
**Query engine:** Flink
**Relevant configs:**
### Logs and Stack Trace
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the Hudi Flink StreamWriteOperatorCoordinator path that recommits an in-flight instant after a restart, and trace how the checkpoint ID is read and written to metadata. Reproduce the storage-failure and recommit sequence described in the issue; done means a checkpoint of -1 is replaced with the previous commit's checkpoint plus one.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100