Refine memory usage of RowContainer
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
[RowContainer](https://github.com/pingcap/tidb/blob/b9a31b231a7d9a64da81cb071b3db26fcb55cc38/pkg/util/chunk/row_container.go#L91) is used for spilling during execution of TiDB operator execution.
But there are lots of temp small objects allocated during spilling. Check [heap_tidb_10.71.222.131_4000_1719544680.txt](https://github.com/user-attachments/files/16023854/heap_tidb_10.71.222.131_4000_1719544680.txt) (PS rename to heap.proto to use it)
The `alloc space` indicates huge number of `chunk.NewReaderWithCache` and `io.NewSectionReader` is called.
But actually after GC, the memory usage is small(check `inuse space` of heap file). **This means we can use cache to reduce the allocation of above small object.**
Following is reproduce sqls:
```
drop table if exists t0;
drop table if exists t1;
create table t0(c1 int);
insert into t0 values (1), (2), (3), (4);
set cte_max_recursion_depth = 10000000;
set tidb_mem_quota_query = 3<<30;
set global tidb_mem_oom_action = 'cancel';
create table t1(c1 int, c2 int);
insert into t1 values(1, 1), (1, 1), (2, 2), (2, 2), (3, 3), (3, 3), (4, 4), (4, 4);
explain analyze with recursive cte1 as (select c1 from t0 union all select cte1.c1 from cte1 inner join t1 on cte1.c1 = t1.c1) select * from cte1;
```
Contributor guide
Assessment
This issue has not been assessed yet.