Make the memory tracking for IndexJoin more accurate.
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
tidb v8.1.1

heap profile
```
----------------------------------------------------------+-------------
flat flat% sum% cum cum% calls calls% + context
----------------------------------------------------------+-------------
1658.23MB 50.68% | github.com/pingcap/tidb/pkg/distsql.CommonHandleRangesToKVRanges /workspace/source/tidb/pkg/distsql/request_builder.go:697 (inline)
1613.72MB 49.32% | github.com/pingcap/tidb/pkg/distsql.CommonHandleRangesToKVRanges /workspace/source/tidb/pkg/distsql/request_builder.go:698 (inline)
3271.95MB 68.35% 68.35% 3271.95MB 68.35% | github.com/pingcap/tidb/pkg/tablecodec.EncodeRowKey /workspace/source/tidb/pkg/tablecodec/tablecodec.go:93
----------------------------------------------------------+-------------
934.10MB 100% | github.com/pingcap/tidb/pkg/executor.(*dataReaderBuilder).buildExecutorForIndexJoinInternal /workspace/source/tidb/pkg/executor/builder.go:4046
934.10MB 19.51% 87.86% 934.10MB 19.51% | github.com/pingcap/tidb/pkg/executor.(*dataReaderBuilder).buildTableReaderForIndexJoin /workspace/source/tidb/pkg/executor/builder.go:4176
```
Explain analyze
```
tidb> desc analyze select /*+ inl_join(t2) */ * from t1 t1 join t1 t2 on t1.a=t2.a and t1.b=t2.b;
+-----------------------------+------------+---------+-----------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+------+
| id | estRows | actRows | task | access object | execution info | operator info | memory | disk |
+-----------------------------+------------+---------+-----------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+------+
| IndexJoin_11 | 2097920.28 | 2097152 | root | | time:1m21.7s, loops:2049, RU:355869.136697, inner:{total:6m41.6s, concurrency:5, task:90, construct:1.63s, fetch:6m39s, build:912.3ms}, probe:851.7ms | inner join, inner:TableReader_10, outer key:test.t1.a, test.t1.b, inner key:test.t1.a, test.t1.b, equal cond:eq(test.t1.a, test.t1.a), eq(test.t1.b, test.t1.b) | 1.41 GB | N/A |
| ├─TableReader_20(Build) | 2097152.00 | 2097152 | root | partition:all | time:53ms, loops:2078, cop_task: {num: 360, max: 1.2s, min: 89.1µs, avg: 26.9ms, p95: 121.4ms, max_proc_keys: 17376, p95_proc_keys: 17376, tot_proc: 1.71s, tot_wait: 136.8ms, copr_cache_hit_ratio: 0.11, build_task_duration: 103.1µs, max_distsql_concurrency: 15}, rpc_info:{Cop:{num_rpc:360, total_time:9.67s}} | data:TableFullScan_19 | 31.1 MB | N/A |
| │ └─TableFullScan_19 | 2097152.00 | 2097152 | cop[tikv] | table:t1 | tikv_task:{proc max:1.14s, min:0s, avg: 12ms, p80:8ms, p95:20ms, iters:3440, tasks:360}, scan_detail: {total_process_keys: 2064320, total_process_keys_size: 578009600, total_keys: 2064642, get_snapshot_time: 10.7ms, rocksdb: {delete_skipped_count: 1718301, key_skipped_count: 3782621, block: {cache_hit_count: 81737, read_count: 5476, read_byte: 3.32 MB, read_time: 14.6ms}}}, time_detail: {total_process_time: 1.71s, total_suspend_time: 2.73s, total_wait_time: 136.8ms, total_kv_read_wall_time: 4.21s, tikv_wall_time: 5.33s} | keep order:false | N/A | N/A |
| └─TableReader_10(Probe) | 2097152.00 | 2097152 | root | partition:all | time:3m43s, loops:2291, cop_task: {num: 8234, max: 2.56s, min: 443.7µs, avg: 336.7ms, p95: 761.1ms, max_proc_keys: 5536, p95_proc_keys: 2016, tot_proc: 16m40.3s, tot_wait: 10m23.8s, copr_cache_hit_ratio: 0.00, build_task_duration: 5.59s, max_distsql_concurrency: 15}, rpc_info:{Cop:{num_rpc:8234, total_time:46m12.4s}} | data:TableRangeScan_9 | N/A | N/A |
| └─TableRangeScan_9 | 2097152.00 | 2097152 | cop[tikv] | table:t2 | tikv_task:{proc max:0s, min:0s, avg: 158.6ms, p80:322.8ms, p95:396.1ms, iters:15305, tasks:8234}, scan_detail: {total_process_keys: 2096891, total_process_keys_size: 587129480, total_keys: 85977880, get_snapshot_time: 868ms, rocksdb: {delete_skipped_count: 228996, key_skipped_count: 2325887, block: {cache_hit_count: 503285435, read_count: 367, read_byte: 227.7 KB, read_time: 3.25ms}}}, time_detail: {total_process_time: 16m40.3s, total_suspend_time: 5m5.4s, total_wait_time: 10m23.8s, total_kv_read_wall_time: 21m44.9s, tikv_wall_time: 35m5.2s} | range: decided by [eq(test.t1.a, test.t1.a) eq(test.t1.b, test.t1.b)], keep order:false | N/A | N/A |
+-----------------------------+------------+---------+-----------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+------+
5 rows in set (1 min 21.77 sec)
```
IndexJoin only tracked 1.4G, but the total memory usage in the heap is around 4G.
Reproduce step:
1. create a cluster table with partition. create table t1(a int, b varchar(750), c int, primary key(a,b,c)) partition by hash(c) partitions 40;
2. insert some values.
3. run indexjoin. See the explain analyze and heap.
Contributor guide
Assessment
This issue has not been assessed yet.