mysql.tidb_runaway_watch_done table grows unboundedly with no GC mechanism
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
1. Start a TiDB cluster (any topology).
2. Create a resource group with `QUERY_LIMIT` or manually add watches via `QUERY WATCH ADD`.
3. Remove watches explicitly via `QUERY WATCH REMOVE ` — each removal inserts a row into `mysql.tidb_runaway_watch_done` (via `handleRunawayWatchDone()` in `pkg/resourcegroup/runaway/record.go:347-381`).
4. Repeat step 2–3 over days/weeks.
5. Observe that `mysql.tidb_runaway_watch_done` grows monotonically:
```sql
SELECT COUNT(*) FROM mysql.tidb_runaway_watch_done;
-- keeps increasing, never decreases
```
### 2. What did you expect to see? (Required)
`mysql.tidb_runaway_watch_done` should have a periodic GC mechanism (similar to `mysql.tidb_runaway_queries`) that cleans up old, already-consumed rows. Once all TiDB nodes' watch-done syncers have advanced their `done_time` checkpoint past a row, that row is no longer needed and should be eligible for cleanup.
### 3. What did you see instead (Required)
`mysql.tidb_runaway_watch_done` has **no cleanup mechanism at all** — rows accumulate indefinitely.
Code-level evidence:
- **Existing GC only covers `tidb_runaway_queries`**: The `runawayRecordGCTicker` in `manager.go:217-218` triggers `deleteExpiredRows()`, but that function hardcodes `tableName = "tidb_runaway_queries"` (`record.go:246`). It does not touch `tidb_runaway_watch_done`.
- **No TTL on table DDL**: The `CREATE TABLE` statement for `tidb_runaway_watch_done` (`bootstrap.go:663`) has no `TTL` clause.
- **`idx_done_time` exists but unused for purge**: The index added by schema upgrade (`bootstrap.go:3323`) is only consumed by the watch-done syncer's checkpoint reads, not by any cleanup logic.
- **Write path has no corresponding delete**: `handleRunawayWatchDone()` (`record.go:347-381`) INSERTs into `watch_done` and DELETEs from `watch`, but nothing ever DELETEs from `watch_done`.
**Important constraint for the fix**: `watch_done` is not just an audit log — the watch-done syncer on each TiDB node reads this table (via `getNewWatchDoneRecords()` using a `done_time`-based cursor) to drive cross-node `removeWatch()` calls. A cleanup mechanism must only delete rows that have been consumed by all nodes' syncers. A safe approach is to retain rows for a configurable duration (e.g. 7 days, far exceeding any realistic syncer lag) and batch-delete older rows, reusing the existing `idx_done_time` index.
### 4. What is your TiDB version? (Required)
Verified on `release-8.5` at commit `6c7aaa0c8d`. Code inspection suggests `master` and all release branches are affected — no GC for this table has ever been implemented.
Contributor guide
Assessment
This issue has not been assessed yet.