pingcap / pingcap/tidb

Runaway query detection (`EXEC_ELAPSED`) intermittently misses queries that stop issuing coprocessor requests before the deadline

Open
#69,254 3 comments 0 reactions 0 assignees View on GitHub
contribution first-time-contributor may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major sig/execution type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report
While testing resource-group runaway queries with `QUERY_LIMIT=(EXEC_ELAPSED ... ACTION=DRYRUN)`,
we observed that a query exceeding the `EXEC_ELAPSED` deadline is only *intermittently* recorded in
`mysql.tidb_runaway_queries`. Debugging with added logs on a local `tiup playground` (TiDB v8.5.x)
revealed the same race also affects `ACTION=KILL` (the query is killed only sometimes), and by code
inspection it affects the other actions (CoolDown, SwitchGroup) as well, since they share the same
detection paths.

### 1. Minimal reproduce step (Required)
Tested on TiDB v8.5.6 (`tiup playground`), single TiDB / TiKV / PD.

```sql
-- Build a table and a CPU-bound (TiDB-side) join query.
CREATE DATABASE IF NOT EXISTS rg_demo;
USE rg_demo;
CREATE TABLE big (id BIGINT PRIMARY KEY AUTO_RANDOM, v BIGINT, pad VARCHAR(200));

SET cte_max_recursion_depth = 50000;
INSERT INTO big (v, pad)
WITH RECURSIVE seq(n) AS (
SELECT 1 UNION ALL SELECT n+1 FROM seq WHERE n < 50000
)
SELECT n, REPEAT('x', 200) FROM seq;

-- Resource group with a time limit just below the query's runtime.
-- The query below runs ~220ms; threshold set to 200ms.
ALTER RESOURCE GROUP default QUERY_LIMIT=(EXEC_ELAPSED='200ms', ACTION=DRYRUN);

-- Run this 10 times. A hash join whose last coprocessor request is dispatched
-- well before the deadline (the post-deadline tail is not on the cop-dispatch path).
SELECT /*+ NO_INDEX_JOIN(a,b) */ COUNT(*)
FROM big a JOIN big b ON a.v % 50 = b.v % 50
WHERE a.pad LIKE '%x%' AND b.pad LIKE '%x%';

-- Wait ~2 mins (4 x flush interval (30s)) for the flush, then check:
SELECT start_time, repeats, action FROM mysql.tidb_runaway_queries
WHERE resource_group_name = 'default' ORDER BY start_time DESC;
```

### 2. What did you expect to see? (Required)
Every run that exceeds `EXEC_ELAPSED` should have its configured action applied deterministically —
recorded in `mysql.tidb_runaway_queries` for DRYRUN, killed for KILL, deprioritized for CoolDown,
moved for SwitchGroup. (All actions also produce a `tidb_runaway_queries` record when detected.)

### 3. What did you see instead (Required)
Only a subset of runs have the action applied. With DRYRUN, in one test 5 runs produced only 3
records; another showed 2 of 5 missed. With KILL, some runs are killed and some complete normally.
The behavior is timing-dependent and non-deterministic. CoolDown and SwitchGroup were not separately
measured but share the same detection paths, so they are expected to miss identically.

### 4. What is your TiDB version? (Required)
| Release Version: 46fcc5c617-dirty
Edition: Community
Git Commit Hash: 46fcc5c617a65b7d822c438281f9109d8cf6d02a
Git Branch: master
UTC Build Time: 2026-06-16 18:52:40
GoVersion: go1.26.4
Race Enabled: false
Check Table Before Drop: false
Store: tikv |

### 5. Root Cause
For a time-based (`EXEC_ELAPSED`) limit there are only two detection paths, and for this query both
miss. **no coprocessor request is dispatched after the deadline** — so the only thing left
watching the post-deadline window is a coarse 100ms background poll:

- `Checker.BeforeCopRequest` (`pkg/resourcegroup/runaway/checker.go`) runs only before each
coprocessor request is dispatched. Instrumented logging shows the last `BeforeCopRequest` for this
query firing with ~65–125ms still left until the deadline, and no further `BeforeCopRequest` calls
after that. So this path never observes `exceedCause != ""`. (Note: this only tells us when cop
requests are *dispatched*; we did not measure when TiKV responded or attribute the post-deadline
tail to a specific phase — only that nothing re-enters the cop-dispatch path after the deadline.)
- `Checker.CheckRuleKillAction` (`checker.go`), driven by the expensive-query handler ticker every
**100ms** (`pkg/util/expensivequery/expensivequery.go:110`), is then the only watcher for the
remainder of the query. When the query overruns the deadline by less than ~one tick interval, it
usually finishes before the ticker samples it → no detection.

`Checker.CheckThresholds` (the coprocessor-response path) also cannot help here: (a) it early-returns
unless `Action == Kill`, and (b) even for KILL it only fires on cop responses — and no response
crosses the deadline in a way that triggers it for this query.

Why KILL *usually* works but seems to fail here: in KILL mode, `BeforeCopRequest` additionally
stamps `req.Context.MaxExecutionDurationMs` on each cop request
(`checker.go`, the `exceedCause == "" && Action == Kill` branch). When a cop request's own execution
on TiKV would exceed the budget, TiKV returns a deadline error that `CheckThresholds` catches
reliably. But that mechanism only fires when an individual cop request is itself long-running past
the deadline. For a query like this — where cop requests are all dispatched before the deadline and
the overrun happens with no further cop dispatch — KILL falls back to the same 100ms-ticker race and
also misses intermittently (observed).

Contributor guide

Open the contributing guide

Research direction

Start with Checker.BeforeCopRequest and Checker.CheckRuleKillAction in pkg/resourcegroup/runaway/checker.go, then inspect the 100ms ticker in pkg/util/expensivequery/expensivequery.go. Reproduce the EXEC_ELAPSED case from the issue and trace detection when no coprocessor request occurs after the deadline. Done means configured DRYRUN, KILL, CoolDown, and SwitchGroup actions are applied deterministically and recorded as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.