planner: push join key filter below window to avoid full-table window computation
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
In the case below, the first query is much slower than the second one, although they have the same semantics.
The second one is faster because it runs the `Join` before the `Window` and avoid the large `Window` computation.
```
EXPLAIN ANALYZE SELECT
t.billcode AS waybill_no,
f.apply_time AS latest_apply_time
FROM demo_site_event t
LEFT JOIN (
SELECT
ft.*,
ROW_NUMBER() OVER (
PARTITION BY ft.waybill_no
ORDER BY ft.apply_time DESC
) AS rn
FROM demo_transfer_event ft
) f
ON t.billcode = f.waybill_no
AND f.rn = 1
WHERE t.scan_time > '2026-01-26 00:00:00'
AND t.scan_time <= '2026-01-27 00:00:00'
AND t.user_code = 'USER_0001'
AND IFNULL(t.sign_flag, 0) = 0
AND IFNULL(t.remark3, 0) = 0
AND IFNULL(t.remark6, 0) = 0;
+------------------------------------+------------+---------+-----------+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| id | estRows | actRows | task | access object | execution info | operator info | memory | disk |
+------------------------------------+------------+---------+-----------+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| HashJoin_17 | 4615.27 | 1000 | root | | time:248.5ms, open:303.7µs, close:18.7µs, loops:5, RU:35.29, build_hash_table:{concurrency:5, time:8.08ms, fetch:7.9ms, max_partition:171.3µs, total_partition:171.3µs, max_build:2.67µs, total_build:2.67µs}, probe:{concurrency:5, time:248.2ms, fetch_and_wait:240.8ms, max_worker_time:248.2ms, total_worker_time:1.24s, max_probe:7.35ms, total_probe:35.6ms, probe_collision:2102} | left outer join, left side:TableReader_37, equal:[eq(test.demo_site_event.billcode, test.demo_transfer_event.waybill_no)] | 72.2 KB | 0 Bytes |
| ├─TableReader_37(Build) | 4590.84 | 1000 | root | | time:7.89ms, open:66.5µs, close:10.9µs, loops:2, cop_task: {num: 3, max: 4.95ms, min: 930.7µs, avg: 2.47ms, p95: 4.95ms, max_proc_keys: 480, p95_proc_keys: 480, tot_proc: 4.09ms, tot_wait: 959.6µs, copr_cache_hit_ratio: 0.33, build_task_duration: 20.9µs, max_distsql_concurrency: 1}, fetch_resp_duration: 7.66ms, rpc_info:{Cop:{num_rpc:3, total_time:7.34ms}} | data:Selection_36 | 51.6 KB | N/A |
| │ └─Selection_36 | 4590.84 | 1000 | cop[tikv] | | tikv_task:{proc max:8ms, min:1ms, avg: 3.33ms, p80:8ms, p95:8ms, iters:22, tasks:3}, scan_detail: {total_process_keys: 704, total_process_keys_size: 57024, total_keys: 706, get_snapshot_time: 309.8µs, rocksdb: {key_skipped_count: 704, block: {cache_hit_count: 10}}}, time_detail: {total_process_time: 4.09ms, total_wait_time: 959.6µs, total_kv_read_wall_time: 2ms, tikv_grpc_process_time: 146.7µs, tikv_grpc_wait_time: 306µs, tikv_wall_time: 5.79ms} | eq(ifnull(test.demo_site_event.remark3, 0), 0), eq(ifnull(test.demo_site_event.remark6, 0), 0), eq(ifnull(test.demo_site_event.sign_flag, 0), 0), eq(test.demo_site_event.user_code, "USER_0001"), gt(test.demo_site_event.scan_time, 2026-01-26 00:00:00.000000), le(test.demo_site_event.scan_time, 2026-01-27 00:00:00.000000) | N/A | N/A |
| │ └─TableFullScan_35 | 11000.00 | 11000 | cop[tikv] | table:t | tikv_task:{proc max:7ms, min:1ms, avg: 3ms, p80:7ms, p95:7ms, iters:22, tasks:3} | keep order:false | N/A | N/A |
| └─Selection_44(Probe) | 804000.00 | 1001000 | root | | time:247.8ms, open:192.5µs, close:5.17µs, loops:979 | eq(Column#16, 1) | 47.2 KB | N/A |
| └─Shuffle_46 | 1005000.00 | 1005000 | root | | time:208.5ms, open:139µs, close:4.88µs, loops:986, ShuffleConcurrency:5 | execution info: concurrency:5, data sources:[TableReader_33] | N/A | N/A |
| └─Window_45 | 1005000.00 | 1005000 | root | | total_time:1.19s, total_open:29.5µs, total_close:2.71µs, loops:989 | row_number()->Column#16 over(partition by test.demo_transfer_event.waybill_no order by test.demo_transfer_event.apply_time desc rows between current row and current row) | N/A | N/A |
| └─Sort_34 | 1005000.00 | 1005000 | root | | total_time:936ms, total_open:25.2µs, total_close:2.21µs, loops:989 | test.demo_transfer_event.waybill_no, test.demo_transfer_event.apply_time:desc | 10.3 MB | 0 Bytes |
| └─ShuffleReceiver_47 | 1005000.00 | 1005000 | root | | total_time:580.4ms, total_open:1.84µs, total_close:333ns, loops:989 | | N/A | N/A |
| └─TableReader_33 | 1005000.00 | 1005000 | root | | time:9.93ms, open:71.3µs, close:833ns, loops:984, cop_task: {num: 44, max: 6.26ms, min: 146.3µs, avg: 833.1µs, p95: 2.46ms, max_proc_keys: 2016, p95_proc_keys: 992, tot_proc: 6.71ms, tot_wait: 10.1ms, copr_cache_hit_ratio: 0.82, build_task_duration: 38.3µs, max_distsql_concurrency: 3}, fetch_resp_duration: 7.98ms, rpc_info:{Cop:{num_rpc:44, total_time:36ms}} | data:TableFullScan_32 | 5.75 MB | N/A |
| └─TableFullScan_32 | 1005000.00 | 1005000 | cop[tikv] | table:ft | tikv_task:{proc max:37ms, min:0s, avg: 10.4ms, p80:14ms, p95:22ms, iters:1154, tasks:44}, scan_detail: {total_process_keys: 7424, total_process_keys_size: 556800, total_keys: 7432, get_snapshot_time: 2.16ms, rocksdb: {key_skipped_count: 7424, block: {cache_hit_count: 52}}}, time_detail: {total_process_time: 6.71ms, total_suspend_time: 182.3µs, total_wait_time: 10.1ms, total_kv_read_wall_time: 4ms, tikv_grpc_process_time: 813.8µs, tikv_grpc_wait_time: 1.92ms, tikv_wall_time: 21.5ms} | keep order:false | N/A | N/A |
+------------------------------------+------------+---------+-----------+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
EXPLAIN ANALYZE WITH filtered_t AS (
SELECT billcode
FROM demo_site_event
WHERE scan_time > '2026-01-26 00:00:00'
AND scan_time <= '2026-01-27 00:00:00'
AND user_code = 'USER_0001'
AND IFNULL(sign_flag, 0) = 0
AND IFNULL(remark3, 0) = 0
AND IFNULL(remark6, 0) = 0
)
SELECT
t.billcode AS waybill_no,
f.apply_time AS latest_apply_time
FROM filtered_t t
LEFT JOIN (
SELECT
ft.waybill_no,
ft.apply_time,
ROW_NUMBER() OVER (
PARTITION BY ft.waybill_no
ORDER BY ft.apply_time DESC
) AS rn
FROM demo_transfer_event ft
WHERE ft.waybill_no IN (SELECT billcode FROM filtered_t)
) f
ON t.billcode = f.waybill_no
AND f.rn = 1;
+--------------------------------------------------+----------+---------+-----------+---------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+---------+
| id | estRows | actRows | task | access object | execution info | operator info | memory | disk |
+--------------------------------------------------+----------+---------+-----------+---------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+---------+
| HashJoin_40 | 4590.84 | 1000 | root | | time:19.4ms, open:126.5µs, close:21.1µs, loops:2, RU:20.65, build_hash_table:{concurrency:5, time:19.1ms, fetch:19ms, max_partition:97.3µs, total_partition:97.3µs, max_build:2.71µs, total_build:2.71µs}, probe:{concurrency:5, time:19.3ms, fetch_and_wait:19.2ms, max_worker_time:19.3ms, total_worker_time:95.8ms, max_probe:120.2µs, total_probe:120.2µs, probe_collision:322} | left outer join, left side:CTEFullScan_54, equal:[eq(test.demo_site_event.billcode, test.demo_transfer_event.waybill_no)] | 64.4 KB | 0 Bytes |
| ├─Selection_55(Build) | 2953.77 | 1000 | root | | time:19.1ms, open:78.8µs, close:15µs, loops:2 | eq(Column#18, 1) | 47.2 KB | N/A |
| │ └─Shuffle_84 | 3692.22 | 5000 | root | | time:18.9ms, open:77.1µs, close:14.6µs, loops:10, ShuffleConcurrency:5 | execution info: concurrency:5, data sources:[Projection_58] | N/A | N/A |
| │ └─Window_56 | 3692.22 | 5000 | root | | total_time:93.5ms, total_open:13.6µs, total_close:3.04µs, loops:13 | row_number()->Column#18 over(partition by test.demo_transfer_event.waybill_no order by test.demo_transfer_event.apply_time desc rows between current row and current row) | N/A | N/A |
| │ └─Sort_83 | 3692.22 | 5000 | root | | total_time:92.6ms, total_open:12µs, total_close:2.5µs, loops:13 | test.demo_transfer_event.waybill_no, test.demo_transfer_event.apply_time:desc | 88.0 KB | 0 Bytes |
| │ └─ShuffleReceiver_86 | 3692.22 | 5000 | root | | total_time:90.8ms, total_open:667ns, total_close:84ns, loops:13 | | N/A | N/A |
| │ └─Projection_58 | 3692.22 | 5000 | root | | time:17.8ms, open:53.9µs, close:10µs, loops:6, Concurrency:5 | test.demo_transfer_event.waybill_no, test.demo_transfer_event.apply_time | 81.2 KB | N/A |
| │ └─IndexHashJoin_66 | 3692.22 | 5000 | root | | time:18.2ms, open:52.8µs, close:5.13µs, loops:6, inner:{total:15.7ms, concurrency:5, task:1, construct:447.7µs, fetch:13.9ms, build:96.1µs, join:1.37ms} | inner join, inner:IndexLookUp_80, outer key:test.demo_site_event.billcode, inner key:test.demo_transfer_event.waybill_no, equal cond:eq(test.demo_site_event.billcode, test.demo_transfer_event.waybill_no) | 612.3 KB | N/A |
| │ ├─HashAgg_75(Build) | 3672.67 | 1000 | root | | time:2.43ms, open:51.5µs, close:3.54µs, loops:7, partial_worker:{wall_time:2.343958ms, concurrency:5, task_num:1, tot_wait:2.036458ms, tot_exec:278.958µs, tot_time:11.592209ms, max:2.320625ms, p95:2.320625ms}, final_worker:{wall_time:2.371958ms, concurrency:5, task_num:5, tot_wait:32.04µs, tot_exec:42ns, tot_time:11.730958ms, max:2.355583ms, p95:2.355583ms} | group by:test.demo_site_event.billcode, funcs:firstrow(test.demo_site_event.billcode)->test.demo_site_event.billcode | 312.4 KB | 0 Bytes |
| │ │ └─Selection_76 | 3672.67 | 1000 | root | | time:2.05ms, open:1.58µs, close:542ns, loops:2 | not(isnull(test.demo_site_event.billcode)) | 28.2 KB | N/A |
| │ │ └─CTEFullScan_77 | 4590.84 | 1000 | root | CTE:filtered_t | time:1.92ms, open:166ns, close:83ns, loops:3 | data:CTE_0 | N/A | N/A |
| │ └─IndexLookUp_80(Probe) | 3692.22 | 5000 | root | | time:13.3ms, open:0s, close:4.17µs, loops:6, index_task: {total_time: 11.6ms, fetch_handle: 11.6ms, build: 2.17µs, wait: 3.08µs}, table_task: {total_time: 2.69ms, num: 2, concurrency: 5}, next: {wait_index: 5.6ms, wait_table_lookup_build: 6.27ms, wait_table_lookup_resp: 1.14ms} | | 368.7 KB | N/A |
| │ ├─IndexRangeScan_78(Build) | 3692.22 | 5000 | cop[tikv] | table:ft, index:idx_waybill_type(waybill_no, apply_type_code) | time:11.5ms, open:0s, close:0s, loops:6, cop_task: {num: 2, max: 6.09ms, min: 4.63ms, avg: 5.36ms, p95: 6.09ms, max_proc_keys: 2984, p95_proc_keys: 2984, tot_proc: 8.39ms, tot_wait: 207.3µs, copr_cache_hit_ratio: 0.00, build_task_duration: 56.5µs, max_distsql_concurrency: 1}, fetch_resp_duration: 11.4ms, rpc_info:{Cop:{num_rpc:2, total_time:10.7ms}}, tikv_task:{proc max:5ms, min:3ms, avg: 4ms, p80:5ms, p95:5ms, iters:13, tasks:2}, scan_detail: {total_process_keys: 5000, total_process_keys_size: 490000, total_keys: 6001, get_snapshot_time: 59.2µs, rocksdb: {key_skipped_count: 5000, block: {cache_hit_count: 4015}}}, time_detail: {total_process_time: 8.39ms, total_suspend_time: 124.2µs, total_wait_time: 207.3µs, total_kv_read_wall_time: 8ms, tikv_grpc_process_time: 294.2µs, tikv_grpc_wait_time: 24.6µs, tikv_wall_time: 9.36ms} | range: decided by [eq(test.demo_transfer_event.waybill_no, test.demo_site_event.billcode)], keep order:false | N/A | N/A |
| │ └─TableRowIDScan_79(Probe) | 3692.22 | 5000 | cop[tikv] | table:ft | total_time:2.26ms, total_open:0s, total_close:2.38µs, loops:7, cop_task: {num: 2, max: 1.11ms, min: 1.09ms, avg: 1.1ms, p95: 1.11ms, max_proc_keys: 2984, p95_proc_keys: 2984, tot_proc: 1.61ms, tot_wait: 94.4µs, copr_cache_hit_ratio: 0.00, build_task_duration: 16.4µs, max_distsql_concurrency: 1}, fetch_resp_duration: 2.21ms, rpc_info:{Cop:{num_rpc:2, total_time:2.2ms}}, tikv_task:{proc max:1ms, min:1ms, avg: 1ms, p80:1ms, p95:1ms, iters:14, tasks:2}, scan_detail: {total_process_keys: 5000, total_process_keys_size: 355000, total_keys: 5003, get_snapshot_time: 63.2µs, rocksdb: {key_skipped_count: 4999, block: {cache_hit_count: 17}}}, time_detail: {total_process_time: 1.61ms, total_wait_time: 94.4µs, total_kv_read_wall_time: 2ms, tikv_grpc_process_time: 28.5µs, tikv_grpc_wait_time: 21.4µs, tikv_wall_time: 1.8ms} | keep order:false | N/A | N/A |
| └─CTEFullScan_54(Probe) | 4590.84 | 1000 | root | CTE:filtered_t AS t | time:2.08ms, open:41.4µs, close:4.25µs, loops:2 | data:CTE_0 | 56.5 KB | 0 Bytes |
| CTE_0 | 4590.84 | 1000 | root | | time:1.92ms, open:166ns, close:83ns, loops:3 | Non-Recursive CTE | N/A | N/A |
| └─TableReader_31(Seed Part) | 4590.84 | 1000 | root | | time:2.04ms, open:37.8µs, close:3.75µs, loops:2, cop_task: {num: 3, max: 1.01ms, min: 202.8µs, avg: 629.7µs, p95: 1.01ms, max_proc_keys: 480, p95_proc_keys: 480, tot_proc: 697.5µs, tot_wait: 187.7µs, copr_cache_hit_ratio: 0.33, build_task_duration: 7.92µs, max_distsql_concurrency: 1}, fetch_resp_duration: 1.95ms, rpc_info:{Cop:{num_rpc:3, total_time:1.85ms}} | data:Projection_21 | 14.1 KB | N/A |
| └─Projection_21 | 4590.84 | 1000 | cop[tikv] | | tikv_task:{proc max:7ms, min:0s, avg: 2.33ms, p80:7ms, p95:7ms, iters:22, tasks:3}, scan_detail: {total_process_keys: 704, total_process_keys_size: 57024, total_keys: 706, get_snapshot_time: 126.3µs, rocksdb: {key_skipped_count: 704, block: {cache_hit_count: 10}}}, time_detail: {total_process_time: 697.5µs, total_wait_time: 187.7µs, tikv_grpc_process_time: 114.2µs, tikv_grpc_wait_time: 40.2µs, tikv_wall_time: 1.21ms} | test.demo_site_event.billcode | N/A | N/A |
| └─Selection_30 | 4590.84 | 1000 | cop[tikv] | | tikv_task:{proc max:7ms, min:0s, avg: 2.33ms, p80:7ms, p95:7ms, iters:22, tasks:3} | eq(ifnull(test.demo_site_event.remark3, 0), 0), eq(ifnull(test.demo_site_event.remark6, 0), 0), eq(ifnull(test.demo_site_event.sign_flag, 0), 0), eq(test.demo_site_event.user_code, "USER_0001"), gt(test.demo_site_event.scan_time, 2026-01-26 00:00:00.000000), le(test.demo_site_event.scan_time, 2026-01-27 00:00:00.000000) | N/A | N/A |
| └─TableFullScan_29 | 11000.00 | 11000 | cop[tikv] | table:demo_site_event | tikv_task:{proc max:7ms, min:0s, avg: 2.33ms, p80:7ms, p95:7ms, iters:22, tasks:3} | keep order:false | N/A | N/A |
+--------------------------------------------------+----------+---------+-----------+---------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+---------+
```
You can use the below SQLs to reproduce this case:
```
CREATE TABLE repro_digit (
n INT PRIMARY KEY
);
INSERT INTO repro_digit VALUES (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
CREATE TABLE repro_seq_1000 (
n INT PRIMARY KEY
);
INSERT INTO repro_seq_1000
SELECT d0.n + d1.n * 10 + d2.n * 100
FROM repro_digit d0
CROSS JOIN repro_digit d1
CROSS JOIN repro_digit d2;
CREATE TABLE demo_site_event (
id BIGINT PRIMARY KEY AUTO_INCREMENT,
billcode VARCHAR(32) NOT NULL,
scan_time DATETIME NOT NULL,
user_code VARCHAR(32) NOT NULL,
sign_flag INT DEFAULT NULL,
remark3 INT DEFAULT NULL,
remark6 INT DEFAULT NULL,
KEY idx_user_scan(user_code, scan_time),
KEY idx_billcode(billcode)
);
CREATE TABLE demo_transfer_event (
id BIGINT PRIMARY KEY AUTO_INCREMENT,
waybill_no VARCHAR(32) NOT NULL,
apply_time DATETIME NOT NULL,
apply_type_code VARCHAR(16) NOT NULL,
payload VARCHAR(64),
KEY idx_waybill_type(waybill_no, apply_type_code)
);
-- Small filtered side: about 1000 rows match the final query.
INSERT INTO demo_site_event(billcode, scan_time, user_code, sign_flag, remark3, remark6)
SELECT
CONCAT('WB', LPAD(n, 8, '0')),
TIMESTAMPADD(SECOND, n % 3600, CAST('2026-01-26 08:00:00' AS DATETIME)),
'USER_0001',
0,
0,
0
FROM repro_seq_1000;
-- Noise on the small side, outside the filter.
INSERT INTO demo_site_event(billcode, scan_time, user_code, sign_flag, remark3, remark6)
SELECT
CONCAT('OTHER', LPAD(a.n * 10 + b.n, 9, '0')),
TIMESTAMPADD(MINUTE, b.n % 1440, CAST('2026-02-01 00:00:00' AS DATETIME)),
'USER_OTHER',
0,
0,
0
FROM repro_seq_1000 a
CROSS JOIN repro_digit b;
-- Large table noise: 1,000,000 unrelated partitions.
INSERT INTO demo_transfer_event(waybill_no, apply_time, apply_type_code, payload)
SELECT
CONCAT('NOISE', LPAD(a.n * 1000 + b.n, 9, '0')),
TIMESTAMPADD(MINUTE, b.n % 1440, CAST('2026-01-01 00:00:00' AS DATETIME)),
CONCAT('T', LPAD(b.n % 20, 2, '0')),
'noise'
FROM repro_seq_1000 a
CROSS JOIN repro_seq_1000 b;
-- Matching transfer rows: only these are useful for the final join.
INSERT INTO demo_transfer_event(waybill_no, apply_time, apply_type_code, payload)
SELECT
CONCAT('WB', LPAD(a.n, 8, '0')),
TIMESTAMPADD(HOUR, b.n, CAST('2026-01-25 00:00:00' AS DATETIME)),
CONCAT('M', LPAD(b.n, 2, '0')),
'match'
FROM repro_seq_1000 a
CROSS JOIN repro_digit b
WHERE b.n < 5;
ANALYZE TABLE demo_site_event, demo_transfer_event;
```
Contributor guide
Research direction
Start by reproducing the provided EXPLAIN ANALYZE query and compare its plan with the faster equivalent described in the issue. Trace the planner's Join, Window, Selection, Sort, and Shuffle operators to find where the join key filter can move below the Window. Done means the plan avoids the full-table window computation while preserving the query's semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100