pingcap / pingcap/tidb

ttl: expose timeout TTL jobs in metrics and Grafana

Open
#69,453 0 comments 0 reactions 1 assignee Claimed by @King-Dylan View on GitHub
severity/minor sig/sql-infra type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Problem

TTL jobs can hit the hard 6-hour timeout and be finished with `summary_text.scan_task_err = "job is timeout"` in `mysql.tidb_ttl_job_history`. However, the current Prometheus/Grafana TTL job status signal does not expose this terminal result.

The `TTL Job Count By Status` panel is backed by `tidb_server_ttl_job_status`, which only reports current in-memory jobs in `running` or `cancelling` status. A timeout job is summarized and removed directly, so `ALL cancelling` stays at 0 and there is no metric series that shows the timeout result.

## Evidence

* `ttlJobTimeout` is currently a fixed 6-hour timeout.
* The timeout path summarizes task results with `scan_task_err = "job is timeout"`, calls job finish, and removes the job.
* `tidb_server_ttl_job_status` is initialized for `running` and `cancelling` only.
* `reportMetrics` counts only `running` and `cancelling` statuses from the in-memory running job map.
* The job history row is currently written as `status = 'finished'`, so the actual timeout result is only discoverable from `summary_text.scan_task_err`.
* `Table Count By TTL Schedule Delay` intentionally filters only successful history rows (`status = 'finished'` and no `scan_task_err`), so it only indirectly shows the impact of timeout by delaying the success watermark.

## Why this matters

Operators can see TTL QPS/processed rows and a current running job, but cannot alert on or dashboard repeated timeout jobs from Prometheus/Grafana. This is confusing when a job is cancelled by the 6-hour timeout but the `cancelling` series remains 0, because that series only represents current jobs in cancelling state, not terminal cancelled/timeout jobs.

## Proposed fix

* Preserve the terminal result in job history by using `timeout` for timeout jobs and `cancelled` for cancellation paths.
* Add a low-cardinality TTL job finish counter, for example `tidb_server_ttl_job_finish_total{result="success|timeout|cancelled"}`. This keeps the additional monitoring storage cost to a few series per TiDB instance.
* Add or adjust Grafana panels so timeout/cancelled terminal results are visible, and rename the existing status panel to clarify that it shows current job status only.

Related historical work:

* #39848 / #39849 added the initial TTL monitor metrics and Grafana charts.
* #42466 / #42468 added TTL schedule delay and insert/delete rows by day.
* #61509 / #61518 fixed empty timeout summaries, but did not add a timeout-result metric or Grafana panel.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.