pingcap / pingcap/tidb

mem leak when kv of one row bigger than 1M

Open
#47,480 4 comments 0 reactions 0 assignees View on GitHub
component/ddl severity/minor type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

currently, lightning using C.alloc to alloc memory for encoded kv, we must free it explicitly

and we alloc bytes buf for encoded kv individually, the minimal size is 1M
https://github.com/pingcap/tidb/blob/fc4f8a1d8f5342cd01f78eb460e47d78d177ed20/br/pkg/lightning/backend/kv/session.go#L113

and we only store one byte buffer for each row
https://github.com/pingcap/tidb/blob/fc4f8a1d8f5342cd01f78eb460e47d78d177ed20/br/pkg/lightning/backend/kv/session.go#L140-L142

if kv is large than 1M, and data and index kv is split in different buf, some buffer will be leaked

to ease reproduce, set this 1M to 10, and add logs on alloc/free
https://github.com/pingcap/tidb/blob/fc4f8a1d8f5342cd01f78eb460e47d78d177ed20/br/pkg/lightning/backend/kv/session.go#L113

import this data
```
/*!40101 SET NAMES binary*/;
INSERT INTO `t` VALUES
(1,aaaaaaaaaaaaaaaaaaaaaaaa),
(2,aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa),
(1,aaaaaaaaaaaaaaaaaaaaaaaa),
(2,aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa),
```
```
/*!40101 SET NAMES binary*/;
CREATE TABLE `t` (
`id` int(11) DEFAULT NULL,
`v` varchar(100) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```

### 2. What did you expect to see? (Required)

### 3. What did you see instead (Required)

### 4. What is your TiDB version? (Required)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.