executor: ordered TableReader can buffer gigabytes of coprocessor responses across partition streams
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
A normal ordered `SELECT` on a large partitioned table can make one `TableReader` retain about 13.9 GiB of coprocessor response memory and be canceled by the TiDB instance memory limit.
All SQL identifiers and example values below are synthetic, and the size figures are rounded. The observed table shape was approximately 646 million rows, 250 bytes average row length, 151 GiB of table data, 161 GiB of indexes, and 32 partitions.
1. Create a 32-partition table with a composite clustered primary key:
Anonymized schema
```sql
CREATE DATABASE IF NOT EXISTS test;
CREATE TABLE test.t (
c01 VARCHAR(64) NOT NULL,
c02 VARCHAR(64) NOT NULL,
c03 VARCHAR(64) NOT NULL,
c04 VARCHAR(64) DEFAULT NULL,
c05 VARCHAR(64) DEFAULT NULL,
c06 VARCHAR(128) NOT NULL,
c07 VARCHAR(32) DEFAULT NULL,
c08 DATETIME DEFAULT NULL,
c09 DATETIME DEFAULT NULL,
c10 BIGINT NOT NULL DEFAULT 0,
c11 TEXT DEFAULT NULL,
c12 DATETIME NOT NULL,
c13 VARCHAR(64) DEFAULT NULL,
c14 VARCHAR(64) DEFAULT NULL,
c15 VARCHAR(64) DEFAULT NULL,
PRIMARY KEY (c02, c03, c12, c06, c01) /*T![clustered_index] CLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
PARTITION BY KEY (c03, c06) PARTITIONS 32;
```
2. Load enough data that each partition spans multiple coprocessor responses. The issue was observed at the scale described above.
3. Use the default paging and DistSQL scan-concurrency settings. The larger query quota below only makes the excessive peak easier to observe before query-level memory handling stops it:
```sql
SET SESSION tidb_enable_paging = ON;
SET SESSION tidb_distsql_scan_concurrency = DEFAULT;
SET SESSION tidb_mem_quota_query = 17179869184;
```
4. Run an ordinary ordered query that TiDB implements as an ordered partitioned `TableReader`:
```sql
SELECT *
FROM test.t
ORDER BY c02, c03, c12, c06, c01;
```
The relevant plan shape is a single visible `TableReader -> TableRangeScan` with `keep order:true`. Internally, TiDB opens multiple independently buffered `SelectResult` streams and merge-sorts them.
### 2. What did you expect to see? (Required)
`TableReader` is a streaming, non-blocking operator. Its memory should remain bounded by a statement-wide in-flight response or byte budget and should apply backpressure to storage when the SQL client consumes rows more slowly than coprocessor workers produce them.
Partition or grouped-range fan-out should not multiply independent response buffers until the operator retains many GiB of coprocessor responses.
### 3. What did you see instead (Required)
The query was canceled with:
```text
Your query has been cancelled due to exceeding the allowed memory limit for the tidb-server instance and this query is currently using the most memory
```
The anonymized runtime evidence was:
```text
TableReader peak memory: 13.9 GiB
Statement peak memory: 14,976,436,975 bytes
Unpacked coprocessor response bytes: 14,970,664,380 bytes
Rows returned before cancellation: 622,592
Rows scanned: 2,933,096
cop_task.num in runtime stats: 43
max_distsql_concurrency: 2
Slow coprocessor response logs: at least 450 across 32 regions
```
The response count, NextGen response-size target, and measured memory reconcile directly:
```text
450 responses * 32 MiB = 15,099,494,400 bytes = 14.0625 GiB
Measured unpacked response bytes = 14,970,664,380 bytes = 13.9425 GiB
Measured statement peak memory = 14,976,436,975 bytes = 13.9479 GiB
Measured average over 450 responses = 31.727 MiB/response
```
The `450` response count is a lower bound because the timing log records only RPCs slower than 300 ms, so `31.727 MiB` is an upper bound on the actual average response size. The NextGen 32 MiB response limit is also a soft target rather than proof that every response is exactly 32 MiB. Even with those caveats, `450 * 32 MiB` is within 0.85% of the measured unpacked response bytes and quantitatively explains the approximately 14 GiB peak.
The statement peak exceeded cumulative unpacked coprocessor response bytes by only 5,772,595 bytes, about 5.5 MiB. This strongly indicates that the response data remained resident in `TableReader` queues or response-backed chunks instead of being bounded by end-to-end backpressure.
The visible `max_distsql_concurrency: 2` is not a statement-wide limit. It is the maximum concurrency of an individual child result stream; runtime-stat merging takes the maximum rather than the aggregate concurrency.
### 4. What is your TiDB version? (Required)
```text
Release Version: v26.3.7
Git Commit Hash: 8c0a0f01cbf6a99b041af30d6d6e3bc27597f5fa
Storage architecture: NextGen
```
### Additional analysis
This appears to be a TiDB-side memory-amplification problem:
1. An ordered partitioned/grouped-range `TableReader` builds one KV request and opens one `SelectResult` for every input stream, then combines them with `sortedSelectResults`: https://github.com/pingcap/tidb/blob/8c0a0f01cbf6a99b041af30d6d6e3bc27597f5fa/pkg/executor/table_reader.go#L439-L467
2. Each child owns an independent `copIterator`, workers, tasks, and response channels. There is no `TableReader`-wide shared request or byte limiter across these children.
3. For an ordered task in v26.3.7, the response channel capacity is 2 without paging but 18 with paging: https://github.com/pingcap/tidb/blob/8c0a0f01cbf6a99b041af30d6d6e3bc27597f5fa/pkg/store/copr/coprocessor.go#L635-L642
4. With 32 active ordered tasks, paging therefore permits up to 576 queued response slots, excluding current cached chunks and blocked workers. The observed response count and memory fit within this window.
5. NextGen can return a continuation range near its response-size limit even when row-count paging is disabled, so setting `tidb_enable_paging=OFF` reduces the TiDB channel capacity from 18 to 2 while storage responses remain segmented: https://github.com/pingcap/tidb/blob/8c0a0f01cbf6a99b041af30d6d6e3bc27597f5fa/pkg/store/copr/coprocessor.go#L1793-L1801
A session-level mitigation is:
```sql
SET SESSION tidb_distsql_scan_concurrency = 1;
SET SESSION tidb_enable_paging = OFF;
```
This reduces workers and queue slots per child, but does not address the missing shared bound across child streams.
A structural fix would add one shared in-flight request/byte limiter across every `SelectResult` owned by a `TableReader`, including partition and grouped-range merge-sort paths. A regression test should verify that increasing the number of partitions/groups does not multiply peak response memory without a statement-wide bound.
This is related in symptom to #16104, which was closed after earlier TableReader rate-limiting work. The current case remains reproducible through multiple independently opened `SelectResult` streams and paging-expanded task buffers.
Contributor guide
Research direction
Read pkg/executor/table_reader.go around the ordered TableReader setup, then inspect pkg/store/copr/coprocessor.go at the response-channel and paging paths. Trace how independently opened SelectResult streams buffer responses, and add a regression test showing that partition or grouped-range fan-out remains within a statement-wide response or byte bound.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100