mem leak when kv of one row bigger than 1M
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
currently, lightning using C.alloc to alloc memory for encoded kv, we must free it explicitly
and we alloc bytes buf for encoded kv individually, the minimal size is 1M
https://github.com/pingcap/tidb/blob/fc4f8a1d8f5342cd01f78eb460e47d78d177ed20/br/pkg/lightning/backend/kv/session.go#L113
and we only store one byte buffer for each row
https://github.com/pingcap/tidb/blob/fc4f8a1d8f5342cd01f78eb460e47d78d177ed20/br/pkg/lightning/backend/kv/session.go#L140-L142
if kv is large than 1M, and data and index kv is split in different buf, some buffer will be leaked
to ease reproduce, set this 1M to 10, and add logs on alloc/free
https://github.com/pingcap/tidb/blob/fc4f8a1d8f5342cd01f78eb460e47d78d177ed20/br/pkg/lightning/backend/kv/session.go#L113
import this data
```
/*!40101 SET NAMES binary*/;
INSERT INTO `t` VALUES
(1,aaaaaaaaaaaaaaaaaaaaaaaa),
(2,aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa),
(1,aaaaaaaaaaaaaaaaaaaaaaaa),
(2,aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa),
```
```
/*!40101 SET NAMES binary*/;
CREATE TABLE `t` (
`id` int(11) DEFAULT NULL,
`v` varchar(100) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```
### 2. What did you expect to see? (Required)
### 3. What did you see instead (Required)
### 4. What is your TiDB version? (Required)
Contributor guide
Assessment
This issue has not been assessed yet.