pingcap / pingcap/tidb

[ANALYZE] Bound sampling memory when statistics underestimate scanned rows

Open
#70,523 1 comment 0 reactions 1 assignee Claimed by @gengliqi View on GitHub
severity/major
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

### 1. Minimal reproduce step (Required)

1. Deploy TiDB `CLOUD.202608.0-5e1901b920` with a 16 GiB memory quota.
2. Start continuous TPCC data loading with 6,000 warehouses and 32 client threads:

```bash
go-tpc tpcc prepare -D tpcc6000 --warehouses 6000 -T 32 --time 36000m --interval 10s
```

3. Keep Auto Analyze enabled while the table is being populated. Reproduce a state where the row-count estimate used to choose the sampling rate is much smaller than the data visible to the Analyze snapshot. In the observed case the chosen sample rate was `0.178` from an estimate of about 620k rows, while the actual data to be sampled was much larger.
4. Let Analyze run concurrently with the TPCC load.

### 2. What did you expect to see? (Required)

Analyze should remain within a bounded memory budget. If its sampling estimate is invalidated by a much larger snapshot, it should reduce/limit sampling or abort Analyze itself. It should not make unrelated foreground SQL statements fail due to the global memory arbitrator.

Image

workload query was be killed
[2026/08/17 20:26:58.318 +08:00] [INFO] [workload.go:709] ["exec statement error: Error 8180 (HY000): Query execution was stopped by the global memory arbitrator [reason=KILL(out-of-memory), path=ParseSQL] [conn=578814022], try again later...\r\n"]

### 3. What did you see instead (Required)

Two continuous-load runs failed when the global memory arbitrator killed a foreground TPCC statement:

- `2026-08-15 13:03:33 +08:00`, TiDB `CLOUD.202608.0-17c0dd0fe4` (`17c0dd0fe42b110ebeacb52f300f742e8e0aad5f`): `Error 8180 (HY000): Query execution was stopped by the global memory arbitrator [reason=KILL(out-of-memory), path=ParseSQL]`.
- `2026-08-17 20:26:58 +08:00`, TiDB `CLOUD.202608.0-5e1901b920` (`5e1901b920a58fce91f8493cfe24299ebe77f168`): same `Error 8180` and `KILL(out-of-memory)` error.

Immediately before the first failure, the TiDB `analyze` memory metric for one TiDB instance peaked at `10,767,274,028` bytes (10.03 GiB); process RSS peaked at `13,407,207,424` bytes (12.49 GiB), against a 16 GiB TiDB memory quota. The second failure showed the same pattern: Analyze about 10.0 GiB and process RSS about 12.5 GiB.

The suspected mechanism is that Bernoulli row sampling uses a fixed sample rate without a fixed retained-sample-count or Analyze-specific memory budget. With a sample rate of 0.178 and an actual scan far larger than the estimate used to choose that rate, the collector can retain millions of rows. The global memory arbitrator acts only as a final safeguard and selected a foreground statement on the `ParseSQL` path as victim rather than stopping the memory-growing Analyze job.

### 4. What is your TiDB version? (Required)

First failing run:

```text
Release Version: CLOUD.202608.0-17c0dd0fe4
Edition: Enterprise
Git Commit Hash: 17c0dd0fe42b110ebeacb52f300f742e8e0aad5f
Git Branch: HEAD
UTC Build Time: 2026-08-14 13:13:49
GoVersion: go1.25.12
Race Enabled: false
Kernel Type: Next Generation
```

Second failing run:

```text
Release Version: CLOUD.202608.0-5e1901b920
Edition: Enterprise
Git Commit Hash: 5e1901b920a58fce91f8493cfe24299ebe77f168
Git Branch: HEAD
UTC Build Time: 2026-08-17 08:40:13
GoVersion: go1.25.12
Race Enabled: false
Kernel Type: Next Generation
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.