executor: SHOW TABLE STATUS can allocate outside the query memory tracker and OOM TiDB
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
The production trigger was the following statement against a database with a very large number of tables, partitions, and statistics records:
```sql
SHOW TABLE STATUS FROM large_schema;
```
A standalone deterministic reproduction script is still needed because the exact customer schema size and deployment details are intentionally omitted. A synthetic reproduction should:
1. Create enough tables and partitions, with statistics, for the table-row statistics cache update to require more memory than the TiDB server budget.
2. Set a finite `tidb_server_memory_limit` and a lower per-query `tidb_mem_quota_query`.
3. Run `SHOW TABLE STATUS FROM large_schema`.
4. Optionally run it through HeidiSQL, or another client that reconnects and retries after a backend network failure, to reproduce the cluster-wide restart loop.
This appears related to, but is not fully covered by, #51260 and #51456 / #51455.
PR #51455 added `memtableRetriever.recordMemoryConsume` for rows returned by information-schema readers. The current path performs additional large allocations outside that accounting:
1. `ShowExec.fetchShowTableStatus` rewrites `SHOW TABLE STATUS` to query `information_schema.tables`, including `TABLE_ROWS`, `AVG_ROW_LENGTH`, `DATA_LENGTH`, and `INDEX_LENGTH`.
2. `memtableRetriever.updateStatsCacheIfNeed` gathers every table and partition ID selected for the database and calls `TableRowStatsCache.UpdateByID`.
3. `UpdateByID` executes restricted SQL against `mysql.stats_meta` and `mysql.stats_histograms`, materializes result slices and maps, and copies them into process-global caches.
4. These restricted-SQL and cache allocations are not fully charged to the initiating statement's memory tracker.
Relevant code:
- `pkg/executor/show.go`: `ShowExec.fetchShowTableStatus`
- `pkg/executor/infoschema_reader.go`: `memtableRetriever.updateStatsCacheIfNeed`
- `pkg/statistics/handle/cache/stats_table_row_cache.go`: `StatsTableRowCache.UpdateByID`, `getRowCountTables`, and `getColLengthTables`
### 2. What did you expect to see? (Required)
- All memory allocated while serving `SHOW TABLE STATUS` should be accounted to a killable query or to the global memory arbitrator.
- If the statement exceeds its memory quota, TiDB should cancel it with an out-of-memory query error.
- The TiDB process should remain alive, and a reconnecting client should not be able to successively terminate every backend.
- The implementation should stream or otherwise bound intermediate information-schema and statistics-cache allocations.
### 3. What did you see instead (Required)
- Go live heap grew from a stable baseline to the container memory boundary within seconds of each statement attempt.
- Live Go objects accounted for about 95% of process RSS, so this was reachable Go heap rather than native memory or RSS-only fragmentation.
- GC ran frequently but could not reclaim the objects while the statement was active.
- The server memory controller repeatedly reported that it could not find an eligible tracked session above `tidb_server_memory_limit_sess_min_size`, even after process heap exceeded `tidb_server_memory_limit`.
- The originating statement's tracked peak remained below the controller's minimum-consumer threshold, which is consistent with the dominant allocations being outside the statement tracker.
- TiDB logs ended abruptly near the container memory boundary, followed by a fresh process start. There was no graceful shutdown, TiDB fatal error, Go runtime OOM message, or intentional TiDB self-exit in the application logs.
- HeidiSQL retried the same statement against successive TiDB backends after receiving backend network failures. Each attempt was followed by the selected backend process restarting. The cycle stopped only after HeidiSQL stopped retrying.
- Other candidate causes, including import, DDL backfill, restore, auto-analyze, plan cache, and ordinary SQL load, were not active or did not correlate with the restarts.
The exact Kubernetes termination reason was not retained in the available telemetry, but the combination of abrupt process-only termination, RSS reaching the fixed container quota, and unaffected sidecars is strongly consistent with a container OOM kill.
### Diagnosis and causal chain
The incident evidence supports the following end-to-end chain:
1. HeidiSQL issued `SHOW TABLE STATUS FROM large_schema` through TiProxy.
2. `ShowExec.fetchShowTableStatus` converted the statement into a restricted query of `information_schema.tables` requesting table-size and row-count columns.
3. Those columns caused `memtableRetriever.updateStatsCacheIfNeed` to enumerate all selected tables and physical partitions and call `TableRowStatsCache.UpdateByID`.
4. `UpdateByID` loaded statistics through restricted SQL, built temporary result slices and maps, and copied the data into process-global caches. The dominant allocations were reachable Go heap but were not fully attributed to the HeidiSQL session's memory tracker.
5. Process heap therefore crossed `tidb_server_memory_limit`, while the global memory controller could not find a sufficiently large tracked session to cancel. GC was active, but the objects remained reachable while the statement was in progress.
6. Process RSS reached the container memory boundary and the TiDB process disappeared abruptly. TiDB did not log a graceful shutdown, panic, fatal error, runtime OOM, or intentional self-exit. TiProxy observed a backend network break/EOF, which is strongly consistent with an external container OOM kill.
7. HeidiSQL automatically reconnected and retried the same statement on another available TiDB backend. The exact statement was observed once for every recorded restart during the incident, creating a repeating query -> heap growth -> process death -> reconnect cycle.
8. After HeidiSQL stopped retrying, newly started TiDB processes remained stable and the restart loop ended.
The internal statistics bootstrap executed during every TiDB startup, but it was not sufficient to cause the large heap ramp: heap remained near the post-start baseline immediately after bootstrap, and the final stable processes executed the same bootstrap successfully. The multi-gigabyte live-heap growth occurred only after the retried `SHOW TABLE STATUS` statement reached a backend.
Confidence is high that the statement and HeidiSQL retry chain caused the restarts. The exact allocation stacks should still be confirmed from an `oom_record` heap profile; the strongest source-correlated candidates are `ShowExec.fetchShowTableStatus`, `memtableRetriever.updateStatsCacheIfNeed`, restricted-SQL result materialization, and `StatsTableRowCache.UpdateByID`.
### 4. What is your TiDB version? (Required)
Observed on the TiDB Cloud release corresponding to `v26.3.4`. Deployment-specific build identifiers are intentionally omitted.
### Additional notes
The earlier fix in #51455 verifies that row materialization in `memtableRetriever` respects `tidb_mem_quota_query`, but it does not appear to cover the restricted-SQL result slices and process-global maps built by `TableRowStatsCache.UpdateByID`.
Suggested regression coverage:
1. Build a database with enough tables, partitions, and statistics rows to exceed a small query quota.
2. Execute `SHOW TABLE STATUS FROM large_schema`.
3. Verify that the statement is cancelled before process RSS approaches the server/container limit.
4. Verify that memory allocated by `UpdateByID`, including temporary maps and restricted-SQL rows, is accounted and released.
5. Verify that repeating the query cannot terminate the TiDB process.
Contributor guide
Research direction
Start with pkg/executor/show.go, pkg/executor/infoschema_reader.go, and pkg/statistics/handle/cache/stats_table_row_cache.go, focusing on the named entry points and allocations. Build the suggested synthetic schema and run SHOW TABLE STATUS with finite memory limits while inspecting heap and memory-tracker behavior. Done means the regression is reproducible and the statement is cancelled within quota without process termination or unbounded cache allocations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- backend, databases, distributed-systems, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100