apache / apache/hudi

Flink Checkpoint -1 is written to hudi metadata

Open
#14,157 2 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**
When Hudi Flink StreamWriteOperatorCoodinator try to recommit the last in flight instant if the write metadata checkpoint successfully but was not committed due to some rare cases, the checkpoint id is set to -1 and committed to metadata

**What you expected:**
When StreamWriteOperatorCoodinator try to commit an instance with checkpoint = -1, we should look up the last commit and find the corresponding checkpoint Id A inside and commit the instant with checkpoint A +1.

**Steps to reproduce:**
1. Flink hudi ingestion job successfully finished checkpoint
2. Hudi Fail due to storage issue
3. Job restart and try to recommit

### Environment

**Hudi version:**
0.14 to main. The behavior is the same.
**Query engine:** Flink
**Relevant configs:**

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the Hudi Flink StreamWriteOperatorCoordinator path that recommits an in-flight instant after a restart, and trace how the checkpoint ID is read and written to metadata. Reproduce the storage-failure and recommit sequence described in the issue; done means a checkpoint of -1 is replaced with the previous commit's checkpoint plus one.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.