apache / apache/seatunnel

[Feature][Zeta] Add a dedicated backpressure diagnostics view for running jobs

Open
#11,352 4 comments 0 reactions 0 assignees View on GitHub
design feature help wanted metrics Zeta Zeta UI
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

## Background
SeaTunnel already exposes some backpressure-related realtime signals, and the current UI can color edges with blocked ratio and queue fill ratio.

That is a good starting point, but it is still not enough for real backpressure diagnosis.

In practice, when a pipeline is slow, users do not just need to know that "an edge looks red". They need to know:
- which edge is the worst bottleneck right now
- whether the bottleneck is stable or transient
- whether it is caused by source production speed, transform CPU cost, sink write/commit pressure, or downstream queue saturation
- whether the pressure is getting worse or recovering

## Problem to solve
Backpressure information currently exists mostly as low-level samples. SeaTunnel still lacks a dedicated user-facing view that turns those samples into an actionable diagnosis workflow.

This leads to slow troubleshooting and makes it hard to compare multiple blocked paths in the same job.

## Proposed scope
Add a dedicated backpressure diagnostics view for running jobs.

Suggested first-delivery scope:
- rank the most blocked edges and downstream bottleneck vertices in one place
- show queue size, queue capacity, fill ratio, blocked ratio, and blocked duration trend together
- distinguish between chronic pressure and short spikes
- provide a direct path from the backpressure panel into the related vertex/edge details and logs
- make the view work together with the runtime graph, but keep it independently useful as a focused bottleneck page

## Why this is not just a UI cleanup
A useful backpressure view may require new or clarified contracts for:
- how blocked time is sampled and aggregated
- whether queue metrics are instantaneous, rolling-window, or max/min summaries
- how long samples are retained
- how high-frequency metrics are cleaned up after job finish/failure
- how to avoid high write amplification for observability-only state

## STIP requirement before implementation
Because this feature may extend realtime metric contracts and has non-trivial runtime-cost implications, **the claimant should submit a STIP design first and get maintainer agreement before starting implementation**.

The STIP should clarify:
- the exact metric model used by the backpressure view
- whether existing metrics are sufficient or new metrics are required
- how the view handles large jobs and fast-refresh scenarios
- how backpressure diagnosis should interact with checkpoint and sink-commit signals
- which data is ephemeral observability state versus stable user-facing API output

## Acceptance criteria
- Users can quickly identify the top blocked edges and likely bottleneck vertices for a running job.
- The view shows both current state and short-term trend, not just a single latest value.
- Data definitions are documented clearly enough that users do not have to infer meaning from implementation details.
- English and Chinese docs are updated.

## Non-goals for the first version
- full root-cause automation for every slowdown
- storage of long-term historical backpressure data
- connector-specific tuning recommendations in the first delivery

## Related work
- PR #11244
- Issue #11351

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing PR #11244 and Issue #11351, then prepare the required STIP design before implementation. Define the metric model, retention and refresh behavior, runtime-cost safeguards, and interactions with checkpoint and sink-commit signals. Done means the agreed view meets the acceptance criteria and English and Chinese documentation is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, frontend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.