apache / apache/seatunnel

[Bug] [imap] engine_runningJobMetrics size too long, imap monitoring data is too large

Open
#8,558 7 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 17h
Merged PRs (30d)
210

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

In split mode, after MySQL CDC runs for a period of time, the imap is stored on the local disk and takes up too much space, a full 200GB. The parameters configured as follows do not clear the data. If this continues, the task will definitely not be able to run.

seatunnel:
engine:
history-job-expire-minutes: 1440
telemetry:
logs:
scheduled-deletion-enable: true

### SeaTunnel Version

2.3.9

### SeaTunnel Config

```conf
MySQL的cdc
```

### Running Command

```shell
MySQL的cdc
```

### Error Exception

```log
MySQL的cdc
```

### Zeta or Flink or Spark Version

_No response_

### Java or Scala Version

_No response_

### Screenshots

![Image](https://github.com/user-attachments/assets/c27bc70f-bcd8-4f93-a40c-91e3d774fca1)

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by investigating the engine telemetry logs and the history-job-expire-minutes and scheduled-deletion-enable settings described in the issue, then reproduce the MySQL CDC split-mode workload. Done means expired monitoring data is deleted as configured and the local disk no longer grows to excessive size.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
backend, databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.