pingcap / pingcap/tidb

MPP response memory is not properly accounted to SQL tracker during MPPDataPacket.Unmarshal

Open
#70,409 2 comments 0 reactions 0 assignees View on GitHub
contribution may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major sig/execution type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

Create a table with many varchar columns and one wide `mediumtext` column:

```sql
CREATE DATABASE IF NOT EXISTS testdb;
USE testdb;

CREATE TABLE `a_table` (
`a1` bigint NOT NULL,
`a2` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
`a3` varchar(30) COLLATE utf8_general_ci DEFAULT NULL,
`a4` varchar(20) COLLATE utf8_general_ci DEFAULT NULL,
`a5` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
`a6` varchar(100) COLLATE utf8_general_ci DEFAULT NULL,
`a7` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
`a8` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
`a9` varchar(30) COLLATE utf8_general_ci NOT NULL,
`a10` int NOT NULL,
`a11` datetime DEFAULT NULL,
`a12` date DEFAULT NULL,
`a13` int DEFAULT NULL,
`a14` varchar(60) COLLATE utf8_general_ci DEFAULT NULL,
`a15` varchar(60) COLLATE utf8_general_ci DEFAULT NULL,
`a16` mediumtext COLLATE utf8_general_ci DEFAULT NULL COMMENT 'wide payload',
`a17` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
`a18` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
`a19` varchar(20) COLLATE utf8_general_ci DEFAULT NULL,
`a20` varchar(5) COLLATE utf8_general_ci DEFAULT NULL,
`a21` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
`a22` varchar(128) COLLATE utf8_general_ci DEFAULT NULL,
`a23` varchar(255) COLLATE utf8_general_ci DEFAULT NULL,
`a24` int NOT NULL DEFAULT '0',
PRIMARY KEY (`a1`) /*T![clustered_index] NONCLUSTERED */,
UNIQUE KEY `uk_a9` (`a9`,`a10`,`a15`,`a4`,`a23`),
KEY `idx_a14` (`a14`),
KEY `idx_a2` (`a2`),
KEY `idx_a3` (`a3`),
KEY `idx_a4` (`a4`),
KEY `idx_a11` (`a11`),
KEY `idx_a12` (`a12`),
KEY `idx_a19_a20` (`a19`,`a20`),
KEY `idx_a22` (`a22`),
KEY `idx_a9` (`a9`),
KEY `idx_a23` (`a23`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;

Load data where:
many rows match a9 LIKE '4%'
column a16 is relatively large, for example about 2 KiB per row
the total matched a16 payload is hundreds of MiB or larger
In my local reproduction, I used 131072 rows and about 256 MiB of total a16 payload.
Add a TiFlash replica and wait until it becomes available:

ALTER TABLE testdb.a_table SET TIFLASH REPLICA 1;

SELECT *
FROM information_schema.tiflash_replica
WHERE TABLE_SCHEMA = 'testdb'
AND TABLE_NAME = 'a_table';

Run the following SQL concurrently, for example in 4 concurrent sessions:

WITH cte1 AS (
SELECT a15, a3, a4, a22, a12,
a13, a11, a9, a23, a5,
a6, a17, a7, a14, a10,
a8, a2, a21, a19, a16
FROM (
SELECT a15, a3, a4, a22, a12,
a13, a11, a9,
IFNULL(a23, '') AS a23, a5,
a6, a7, a17, a14, a10,
a8, a2, a21, a19, a16,
ROW_NUMBER() OVER (
PARTITION BY a9, a4, a15, a23
ORDER BY a11 DESC
) AS rn
FROM testdb.a_table
WHERE a9 LIKE CONCAT('4', '%')
) sub
),
cte2 AS (
SELECT a9, a23, a4, MIN(a13) AS a13
FROM cte1
GROUP BY a4, a9, a23
HAVING MIN(a13) = NULL
)
SELECT COUNT(*) AS total
FROM cte1;

The execution plan uses TiFlash MPP and has the following shape:

HashAgg
└─CTEFullScan
CTE_0
└─TableReader
└─ExchangeSender
└─Projection
└─Window(row_number)
└─Sort
└─ExchangeReceiver
└─ExchangeSender(HashPartition)
└─Projection
└─TableFullScan pushed down to TiFlash

While the SQL is running, check TiDB process memory, SQL memory tracker, and heap profile.
For example:

Collect the TiDB heap profile from the status port:

### 2. What did you expect to see? (Required)

Collect the TiDB heap profile from the status port:

### 3. What did you see instead (Required)

TiDB process memory can grow very high during concurrent MPP execution and can reach tidb_server_memory_limit.
However, TiDB SQL memory tracking does not show a SQL consuming comparable memory. In my reproduction:
TiDB RSS peak was about 3.31 GiB
the largest single SQL memory shown by processlist was only about 617.8 MiB
after the queries finished, SQL tracker memory returned to 0
heap profile showed most live heap under MPP packet unmarshalling:
668.63MB github.com/pingcap/kvproto/pkg/mpp.(*MPPDataPacket).Unmarshal
697.40MB cumulative in gRPC / TiKV MPP receive path
This suggests that MPP response memory is allocated before it is charged to the SQL tracker.
The suspected code path is:

google.golang.org/grpc.recv()
-> codec.Unmarshal()
-> github.com/pingcap/kvproto/pkg/mpp.(*MPPDataPacket).Unmarshal()
-> MPPDataPacket.Data allocation/copy

TiDB only charges memory later in localMppCoordinator.sendToRespCh. Also, sendToRespCh releases the charge too early by using defer Consume(-respSize) after sending the response to respChan.
Downstream, selectResult.fetchResp calls tipb.SelectResponse.Unmarshal(resultSubset.GetData()), which can allocate more memory before charging the decoded SelectResponse.
Relevant code in v8.5.2:
pkg/executor/internal/mpp/local_mpp_coordinator.go
sendToRespCh(): charges resp.MemSize(), then defers uncharge
receiveResults(): calls stream.Recv()
handleMPPStreamResponse(): wraps MPPDataPacket as mppResponse

pkg/distsql/select_result.go
fetchResp(): calls resultSubset.GetData(), then SelectResponse.Unmarshal(), then charges memory

github.com/pingcap/kvproto/pkg/mpp/mpp.pb.go
MPPDataPacket.Unmarshal(): m.Data = append(m.Data[:0], ...)
This makes the global memory controller observe high Go heap usage, while MemUsageTop1Tracker may not identify the real SQL memory consumer.

### 4. What is your TiDB version? (Required)

Release Version: v8.5.2
Git Commit Hash: f43a13324440f92209e2a9f04c0bbe9cf763978d

Contributor guide

Open the contributing guide

Research direction

Read pkg/executor/internal/mpp/local_mpp_coordinator.go, especially receiveResults, handleMPPStreamResponse, and sendToRespCh, then trace pkg/distsql/select_result.go fetchResp and MPPDataPacket.Unmarshal in mpp.pb.go. Done means MPP response allocations and decoded SelectResponse memory are accounted for through the SQL tracker for the reported concurrent workload, without charges being released before downstream consumption.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.