apache / apache/seatunnel

[Improve][Zeta] Make metrics retrieval fail fast when worker nodes are unhealthy

Open
#11,587 2 comments 0 reactions 0 assignees View on GitHub
help wanted improve metrics Zeta
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

## Background

In separated Zeta clusters, Worker nodes run as Hazelcast Lite Members and should not own IMap partitions. However, a production Kubernetes case showed that when a Worker becomes unhealthy because of memory pressure, long GC pauses, or OOM risk, metrics-related REST/control-plane requests can still be amplified by synchronous RPC waits before Hazelcast removes the stale member.

The root cause of that production incident was the Worker JVM/container memory layout, not Separated Cluster Mode itself. After reducing the Worker heap and leaving enough native-memory headroom, the cluster recovered. However, the case exposed a hardening gap in the metrics retrieval path.

## Current behavior

Several metrics retrieval paths synchronously wait for Worker-side operations:

- `JobMaster#getCurrJobMetrics()` sends `GetTaskGroupMetricsOperation` to Worker nodes and waits on the invocation future.
- `CoordinatorService#getRunningJobMetrics()` sends `GetMetricsOperation` to Worker nodes and waits on the invocation future.
- REST job-info / running metrics requests can therefore wait behind an unhealthy or slow Worker while the member is still visible in Hazelcast membership.

When one Worker is experiencing long GC pauses, native-memory pressure, or delayed response, metrics and job-info requests can become slow or blocked, even though the control plane should ideally remain available with degraded metrics.

## Expected behavior

Metrics retrieval should be best-effort and bounded:

1. Use an explicit request deadline for metrics collection.
2. Query Workers concurrently rather than serially where possible.
3. Return partial metrics when some Workers are unavailable or stale.
4. Mark missing Worker metrics as unavailable/stale instead of blocking the whole response.
5. Keep job status and control-plane metadata available even if live Worker metrics cannot be collected.
6. Avoid relying only on global Hazelcast operation timeouts, because those timeouts also affect other engine operations.

## Suggested scope

- Add bounded timeout handling around Worker metrics RPCs.
- Avoid serial unbounded waits across Workers.
- Preserve existing metrics JSON compatibility as much as possible.
- Add tests for one unreachable/slow Worker not blocking the whole metrics response.
- Document the degraded/partial metrics behavior if the response shape changes.

## Related but separate work

This is different from:

- #10997 / #11564, which focus on offloading `ReportMetricsOperation` from Hazelcast generic-operation threads.
- #11472, which focuses on inaccurate finished job metrics after worker failover.
- #10808, which focuses on routing control-plane operations to coordinator-capable nodes in separated mode.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with JobMaster#getCurrJobMetrics() and CoordinatorService#getRunningJobMetrics(), tracing GetTaskGroupMetricsOperation and GetMetricsOperation invocation futures. Done means bounded concurrent collection returns partial metrics while job status and control-plane metadata remain available; add coverage for an unreachable or slow Worker and document any response-shape change.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.