Runtime stats formatting may panic in tdigest.Quantile with index out of range
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
This was observed with a high-fan-out distributed query which produced more than 12,000 Cop tasks. When the query finished, TiDB tried to reset the connection process info to `ComSleep`. While generating the brief binary plan, runtime-stat formatting calculated the p95 Cop response time through `execdetails.Percentile`, which had switched to `influxdata/tdigest` after collecting more than 1,000 samples.
A deterministic reproducer has not been minimized yet. A likely reproducer is to concurrently update and stringify a `selectResultRuntimeStats`/`Percentile` after it has entered the TDigest path.
Observed stack:
```text
runtime error: index out of range [868] with length 868
github.com/influxdata/tdigest.(*TDigest).Quantile
github.com/influxdata/tdigest@v0.0.1/tdigest.go:182
github.com/pingcap/tidb/pkg/util/execdetails.(*Percentile[...]).GetPercentile
pkg/util/execdetails/util.go:233
github.com/pingcap/tidb/pkg/distsql.(*selectResultRuntimeStats).String
pkg/distsql/select_result.go:1108
github.com/pingcap/tidb/pkg/planner/core.binaryOpFromFlatOp
github.com/pingcap/tidb/pkg/planner/core.binaryOpTreeFromFlatOps
github.com/pingcap/tidb/pkg/planner/core.binaryDataFromFlatPlan
github.com/pingcap/tidb/pkg/planner/core.BinaryPlanStrFromFlatPlan
github.com/pingcap/tidb/pkg/planner/core.GetBriefBinaryPlan
github.com/pingcap/tidb/pkg/session.(*session).SetProcessInfo
github.com/pingcap/tidb/pkg/server.(*clientConn).dispatch.func2
github.com/pingcap/tidb/pkg/server.(*clientConn).dispatch
github.com/pingcap/tidb/pkg/server.(*clientConn).Run
```
### 2. What did you expect to see? (Required)
Runtime-stat formatting and process-info cleanup should never panic, regardless of the number of Cop tasks or concurrent stats updates.
### 3. What did you see instead (Required)
`TDigest.Quantile(0.95)` called `sort.Search` over `t.cumulative`; it returned `lower == len(t.cumulative)` (868), then line 182 accessed `t.cumulative[lower]` and panicked. The panic was recovered by the per-connection running loop, so the TiDB process stayed alive, but the client connection/query failed.
For a consistent TDigest, the final cumulative value should equal `processedWeight`, so a p95 search should not return `len`. This suggests the TDigest state became inconsistent, most likely because runtime stats were read while they were still being updated. `execdetails.Percentile` has no synchronization, and `RuntimeStatsColl.GetRootStats` returns an internal pointer after releasing its mutex before `String()` traverses the stats.
The dependency also assumes its internal invariant and has no guard for `lower == len(t.cumulative)`, turning an inconsistent snapshot into an out-of-bounds panic.
Possible fixes:
1. Snapshot or lock runtime stats while generating the brief binary plan / calling `String()`.
2. Make `Percentile` safe against concurrent `Add`/`MergePercentile` and `GetPercentile`, or ensure ownership prevents such concurrency.
3. Add a defensive boundary check in the TDigest quantile path (or replace/update the dependency), so process-info formatting cannot panic even if the invariant is violated.
4. Add a stress/race test with more than 1,000 samples and concurrent stats formatting.
### 4. What is your TiDB version? (Required)
```text
Release Version: CLOUD.202603.10
TiDB Component Version: v26.3.10
Git Commit Hash: 6b4c9f9dce2118aca580a16ef49bd791f4e12b3a
Go Version: go1.25.12
```
Contributor guide
Research direction
Start in pkg/util/execdetails/util.go and pkg/distsql/select_result.go, then trace RuntimeStatsColl.GetRootStats and the SetProcessInfo path shown in the stack. Run or add a race/stress test using more than 1,000 samples and concurrent formatting. Done means p95 formatting and process-info cleanup no longer expose a panic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100