apache / apache/hudi

Flink recommit can fail when the initial heartbeat write times out

Open
#19,947 0 comments 0 reactions 1 assignee Claimed by @cshuo View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Description

During Flink recovery, recommitting a pending instant can fail if the initial heartbeat write times out and commit proceeds before a background retry succeeds.

`HoodieHeartbeatClient.start()` attempts the first heartbeat write synchronously. On timeout, `updateHeartbeat()` logs the failure without recording a successful heartbeat. `start()` then schedules background retries and returns normally.

This behavior intentionally allows recovery from transient storage delays. However, Flink recommit proceeds directly to commit using restored write statuses, leaving little time for a background retry to succeed.

### Failure sequence

1. Flink restores write metadata for a pending instant.
2. `recommitInstant()` calls `restartHeartbeat()`.
3. The initial data-table or metadata-table heartbeat write times out.
4. `start()` returns normally, with background retries scheduled.
5. Recommit reaches heartbeat validation before a retry succeeds.
6. If no heartbeat file is available, the check reads `0` and rejects the commit as expired.

### Observed error

```text
HoodieException: Failed to start operator coordinator.
Caused by: HoodieException: Error while completing streaming commit
to metadata with instant 20260831111848938
Caused by: HoodieException: Heartbeat for instant 20260831111848938
has expired, last heartbeat 0
```

Relevant call path:

```text
StreamWriteOperatorCoordinator.start
→ restoreEvents
→ recommitInstant
→ commitInstant
→ commitToMetadataTable
→ completeStreamingCommit
→ commitStats
→ abortIfHeartbeatExpired
```

The initial heartbeat write timeout was confirmed in the reported case. The reason the previous heartbeat file was absent has not been established.

### Expected behavior

Before recommitting, Flink should allow a bounded opportunity for the required heartbeats to become available. If they remain unavailable, recovery should fail with an explicit heartbeat-readiness timeout.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.