[Improve][Zeta] Make metrics retrieval fail fast when worker nodes are unhealthy
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 204
Description
## Background
In separated Zeta clusters, Worker nodes run as Hazelcast Lite Members and should not own IMap partitions. However, a production Kubernetes case showed that when a Worker becomes unhealthy because of memory pressure, long GC pauses, or OOM risk, metrics-related REST/control-plane requests can still be amplified by synchronous RPC waits before Hazelcast removes the stale member.
The root cause of that production incident was the Worker JVM/container memory layout, not Separated Cluster Mode itself. After reducing the Worker heap and leaving enough native-memory headroom, the cluster recovered. However, the case exposed a hardening gap in the metrics retrieval path.
## Current behavior
Several metrics retrieval paths synchronously wait for Worker-side operations:
- `JobMaster#getCurrJobMetrics()` sends `GetTaskGroupMetricsOperation` to Worker nodes and waits on the invocation future.
- `CoordinatorService#getRunningJobMetrics()` sends `GetMetricsOperation` to Worker nodes and waits on the invocation future.
- REST job-info / running metrics requests can therefore wait behind an unhealthy or slow Worker while the member is still visible in Hazelcast membership.
When one Worker is experiencing long GC pauses, native-memory pressure, or delayed response, metrics and job-info requests can become slow or blocked, even though the control plane should ideally remain available with degraded metrics.
## Expected behavior
Metrics retrieval should be best-effort and bounded:
1. Use an explicit request deadline for metrics collection.
2. Query Workers concurrently rather than serially where possible.
3. Return partial metrics when some Workers are unavailable or stale.
4. Mark missing Worker metrics as unavailable/stale instead of blocking the whole response.
5. Keep job status and control-plane metadata available even if live Worker metrics cannot be collected.
6. Avoid relying only on global Hazelcast operation timeouts, because those timeouts also affect other engine operations.
## Suggested scope
- Add bounded timeout handling around Worker metrics RPCs.
- Avoid serial unbounded waits across Workers.
- Preserve existing metrics JSON compatibility as much as possible.
- Add tests for one unreachable/slow Worker not blocking the whole metrics response.
- Document the degraded/partial metrics behavior if the response shape changes.
## Related but separate work
This is different from:
- #10997 / #11564, which focus on offloading `ReportMetricsOperation` from Hazelcast generic-operation threads.
- #11472, which focuses on inaccurate finished job metrics after worker failover.
- #10808, which focuses on routing control-plane operations to coordinator-capable nodes in separated mode.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with JobMaster#getCurrJobMetrics() and CoordinatorService#getRunningJobMetrics(), tracing GetTaskGroupMetricsOperation and GetMetricsOperation invocation futures. Done means bounded concurrent collection returns partial metrics while job status and control-plane metadata remain available; add coverage for an unreachable or slow Worker and document any response-shape change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100