pingcap / pingcap/tidb

Refine memory usage of RowContainer

Open
#54,294 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
[RowContainer](https://github.com/pingcap/tidb/blob/b9a31b231a7d9a64da81cb071b3db26fcb55cc38/pkg/util/chunk/row_container.go#L91) is used for spilling during execution of TiDB operator execution.

But there are lots of temp small objects allocated during spilling. Check [heap_tidb_10.71.222.131_4000_1719544680.txt](https://github.com/user-attachments/files/16023854/heap_tidb_10.71.222.131_4000_1719544680.txt) (PS rename to heap.proto to use it)

The `alloc space` indicates huge number of `chunk.NewReaderWithCache` and `io.NewSectionReader` is called.
image

But actually after GC, the memory usage is small(check `inuse space` of heap file). **This means we can use cache to reduce the allocation of above small object.**
image

Following is reproduce sqls:
```
drop table if exists t0;
drop table if exists t1;
create table t0(c1 int);
insert into t0 values (1), (2), (3), (4);
set cte_max_recursion_depth = 10000000;
set tidb_mem_quota_query = 3<<30;
set global tidb_mem_oom_action = 'cancel';
create table t1(c1 int, c2 int);
insert into t1 values(1, 1), (1, 1), (2, 2), (2, 2), (3, 3), (3, 3), (4, 4), (4, 4);
explain analyze with recursive cte1 as (select c1 from t0 union all select cte1.c1 from cte1 inner join t1 on cte1.c1 = t1.c1) select * from cte1;
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.