pingcap / pingcap/tidb

Runtime stats formatting may panic in tdigest.Quantile with index out of range

Open
#70,655 1 comment 0 reactions 0 assignees View on GitHub
affects-26.3 component/server component/util may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

This was observed with a high-fan-out distributed query which produced more than 12,000 Cop tasks. When the query finished, TiDB tried to reset the connection process info to `ComSleep`. While generating the brief binary plan, runtime-stat formatting calculated the p95 Cop response time through `execdetails.Percentile`, which had switched to `influxdata/tdigest` after collecting more than 1,000 samples.

A deterministic reproducer has not been minimized yet. A likely reproducer is to concurrently update and stringify a `selectResultRuntimeStats`/`Percentile` after it has entered the TDigest path.

Observed stack:

```text
runtime error: index out of range [868] with length 868

github.com/influxdata/tdigest.(*TDigest).Quantile
github.com/influxdata/tdigest@v0.0.1/tdigest.go:182
github.com/pingcap/tidb/pkg/util/execdetails.(*Percentile[...]).GetPercentile
pkg/util/execdetails/util.go:233
github.com/pingcap/tidb/pkg/distsql.(*selectResultRuntimeStats).String
pkg/distsql/select_result.go:1108
github.com/pingcap/tidb/pkg/planner/core.binaryOpFromFlatOp
github.com/pingcap/tidb/pkg/planner/core.binaryOpTreeFromFlatOps
github.com/pingcap/tidb/pkg/planner/core.binaryDataFromFlatPlan
github.com/pingcap/tidb/pkg/planner/core.BinaryPlanStrFromFlatPlan
github.com/pingcap/tidb/pkg/planner/core.GetBriefBinaryPlan
github.com/pingcap/tidb/pkg/session.(*session).SetProcessInfo
github.com/pingcap/tidb/pkg/server.(*clientConn).dispatch.func2
github.com/pingcap/tidb/pkg/server.(*clientConn).dispatch
github.com/pingcap/tidb/pkg/server.(*clientConn).Run
```

### 2. What did you expect to see? (Required)

Runtime-stat formatting and process-info cleanup should never panic, regardless of the number of Cop tasks or concurrent stats updates.

### 3. What did you see instead (Required)

`TDigest.Quantile(0.95)` called `sort.Search` over `t.cumulative`; it returned `lower == len(t.cumulative)` (868), then line 182 accessed `t.cumulative[lower]` and panicked. The panic was recovered by the per-connection running loop, so the TiDB process stayed alive, but the client connection/query failed.

For a consistent TDigest, the final cumulative value should equal `processedWeight`, so a p95 search should not return `len`. This suggests the TDigest state became inconsistent, most likely because runtime stats were read while they were still being updated. `execdetails.Percentile` has no synchronization, and `RuntimeStatsColl.GetRootStats` returns an internal pointer after releasing its mutex before `String()` traverses the stats.

The dependency also assumes its internal invariant and has no guard for `lower == len(t.cumulative)`, turning an inconsistent snapshot into an out-of-bounds panic.

Possible fixes:

1. Snapshot or lock runtime stats while generating the brief binary plan / calling `String()`.
2. Make `Percentile` safe against concurrent `Add`/`MergePercentile` and `GetPercentile`, or ensure ownership prevents such concurrency.
3. Add a defensive boundary check in the TDigest quantile path (or replace/update the dependency), so process-info formatting cannot panic even if the invariant is violated.
4. Add a stress/race test with more than 1,000 samples and concurrent stats formatting.

### 4. What is your TiDB version? (Required)

```text
Release Version: CLOUD.202603.10
TiDB Component Version: v26.3.10
Git Commit Hash: 6b4c9f9dce2118aca580a16ef49bd791f4e12b3a
Go Version: go1.25.12
```

Contributor guide

Open the contributing guide

Research direction

Start in pkg/util/execdetails/util.go and pkg/distsql/select_result.go, then trace RuntimeStatsColl.GetRootStats and the SetProcessInfo path shown in the stack. Run or add a race/stress test using more than 1,000 samples and concurrent formatting. Done means p95 formatting and process-info cleanup no longer expose a panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.