apache / apache/seatunnel

[Improve][Zeta] Add historical task metrics persistence and time-range query support

Open
#11,784 5 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues) and found no similar feature request.

### Motivation

SeaTunnel already has runtime metrics and realtime observability for running jobs, but it does not yet provide a complete mechanism for **historical task metrics persistence** and **time-range querying**.

At the moment, the gap is mainly around questions like:

- How many records were read or written during a specific time range?
- What was the throughput trend of a source / transform / sink task over time?
- What sampling interval is used for metric collection?
- Where are these sampled metrics persisted for later analysis?

This is especially important for long-running streaming and CDC jobs, where users often need to investigate performance regressions, backpressure, throughput drops, and task-level bottlenecks after the fact.

### Current limitation

From the current implementation, SeaTunnel Zeta mainly provides:

- latest task metrics snapshots
- short-window realtime observability
- final job metrics after job completion

However, it does **not** provide a built-in solution for:

- persistent storage of task/operator historical metrics
- configurable sampling interval for historical collection
- querying metrics by time range
- a clear storage abstraction for historical metrics backend
- REST/UI support for historical metric exploration

### Expected behavior

It would be useful to support a historical metrics mechanism for Zeta, for example:

1. Configurable metric sampling interval
2. Configurable retention period
3. Persistent storage backend for historical metrics
4. Query API by:
- job
- pipeline
- task / vertex / operator
- metric name
- time range
5. Optional UI support for trend charts and time-series inspection

### Possible design direction

A possible direction could be:

- keep the current realtime observability path lightweight
- introduce a separate historical metrics export/persistence path
- support pluggable backends such as:
- Prometheus-compatible remote storage
- OpenTelemetry metrics pipeline
- InfluxDB
- other external TSDB systems
- avoid coupling historical metrics storage with checkpoint state

This would keep checkpoint/recovery semantics clean while making observability more production-friendly.

### Use cases

- Investigate how many records were read in the last 10 minutes / 1 hour
- Compare source and sink throughput over a time range
- Analyze historical backpressure trends
- Troubleshoot intermittent performance issues after the job has recovered or restarted
- Build dashboards and alerting on SeaTunnel task-level metrics

### Looking for contributors

I believe this would be a valuable improvement for SeaTunnel Zeta observability.

If the community agrees with this direction, I would like to ask:

- Is there already a related design or roadmap?
- Would maintainers be open to a proposal for this feature?
- Are there contributors interested in collaborating on the design and implementation?

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named in the issue. Start by reviewing SeaTunnel Zeta's existing runtime metrics and realtime observability paths, then define the sampling, retention, persistence, time-range query, and REST/UI boundaries. Done means the community has an agreed design and implementation scope for historical task metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, prometheus
Domain
backend, distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.