apache / apache/hudi

Add coordinator metric for pending commit instants in Flink streaming write

Open
#19,116 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Background

In Flink streaming writes, the `StreamWriteOperatorCoordinator` buffers `WriteMetadataEvent`s from write tasks per checkpoint. Once all events for a checkpoint are received (the snapshot/data write of that batch has succeeded on the task side), the corresponding instant becomes a *completed but not-yet-committed* instant — it is waiting for the coordinator to commit it on checkpoint completion.

Today there is no easy way to observe how many such instants are pending commit at the coordinator, operators have no direct signal and must infer the problem from logs or the timeline.

## Proposal

Expose a coordinator-level metric `pendingCommitInstantCount` that reports the number of instants whose data writes have completed (snapshot succeeded) but which have not yet been committed by the coordinator.

This makes it straightforward to monitor, at a glance, how many completed-but-uncommitted instants are currently accumulating, helping diagnose commit backpressure and stalls in the streaming write pipeline.

## Notes

- Implemented as a Flink gauge registered by the coordinator on start.
- Empty data-write buffers are excluded from the count so they are not reported as pending commits.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at StreamWriteOperatorCoordinator and inspect its start lifecycle and per-checkpoint data-write buffers. Verify that the coordinator-level gauge counts completed, non-empty buffers awaiting commit, excludes empty data-write buffers, and reflects commits removing pending instants.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.