apache / apache/fluss

[server] Add coordinator metrics for rebalance

Open
#4,055 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

### Motivation

Rebalance currently exposes no metrics at all. The only way to observe a rebalance is to poll `Admin#listRebalanceProgress` or tail coordinator logs. #3965 (PR #3985) improves the Admin API with timestamps and historical listing, but that covers on-demand queries, not continuous monitoring. Operators cannot:

- Alert when a rebalance is running, stuck, or failing.
- Chart migration throughput and per-bucket duration on a dashboard.
- Correlate a rebalance window with write-latency or disk-usage changes on TabletServers.

The coordinator already registers cluster gauges such as `bucketCount` through `CoordinatorMetricGroup` (`fluss-server/.../metrics/group/CoordinatorMetricGroup.java`), and `RebalanceManager` already tracks all the needed state (`inProgressRebalanceTasksQueue`, `inProgressRebalanceTasks`, `finishedRebalanceTasks`, the in-flight task start timestamp), so this is a wiring task, not a new subsystem.

### Proposed solution

Register a rebalance metric set on the coordinator metric group, with names added to `MetricNames`:

Gauges for the current task:

- `rebalanceInProgress` (0 or 1)
- `rebalanceBucketsPending`
- `rebalanceBucketsCompleted`, `rebalanceBucketsFailed`, `rebalanceBucketsTimedOut`
(split of the finished map by per-bucket status)
- `rebalanceDurationMs` (time since the task was registered, 0 when idle)
- `inflightBucketDurationMs` (time the current bucket task has been running, 0 when idle)

Counters across tasks (survive task completion, reset on coordinator failover):

- `rebalancesCompletedTotal`, `rebalancesFailedTotal`, `rebalancesCanceledTotal`

Implementation notes:

- Expose the values from `RebalanceManager` via supplier gauges. The backing maps are `ConcurrentHashMap` and the in-flight fields are volatile, so reads from the metrics reporter thread are safe without touching the coordinator event thread.
- Register on `CoordinatorMetricGroup`, following the existing `bucketCount` pattern.
- Document the new metrics in `website/docs/maintenance/observability/monitor-metrics.md`.

Out of scope: per-table or per-bucket metric groups (cardinality), and the Admin API side already covered by #3965.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with fluss-server/.../metrics/group/CoordinatorMetricGroup.java and its existing bucketCount registration, then inspect RebalanceManager and MetricNames for the tracked state and metric naming. Document the metrics in website/docs/maintenance/observability/monitor-metrics.md. Done means the coordinator exposes the requested gauges and counters with the stated idle, status, and failover behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.