The plan of TPC-C query is not stable
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
The plan of two queries in TPC-C `ORDER_STATUS` transaction is not stable:
# First Query
```
SELECT c_balance, c_first, c_middle, c_id FROM customer WHERE c_w_id = ? AND c_d_id = ? AND c_last = ? ORDER BY c_first
```
Wrong plan:
```
| id | estRows | estCost | actRows | task | access object | execution info | operator info | memory | disk |
| Sort_5 | 224.70 | 382240.44 | 3 | root | | time:9.41ms, loops:2 | tpcc.customer.c_first | 1.89 KB | 0 Bytes |
| └─Projection_7 | 224.70 | 291683.87 | 3 | root | | time:9.37ms, loops:2, Concurrency:OFF | tpcc.customer.c_balance, tpcc.customer.c_first, tpcc.customer.c_middle, tpcc.customer.c_id | 3.21 KB | N/A |
| └─TableReader_10 | 224.70 | 291594.17 | 3 | root | | time:9.37ms, loops:2, cop_task: {num: 1, max: 9.32ms, proc_keys: 3000, tot_proc: 8.93ms, tot_wait: 35.8µs, rpc_num: 1, rpc_time: 9.29ms, copr_cache_hit_ratio: 0.00, build_task_duration: 4.8µs, max_distsql_concurrency: 1} | data:Selection_9 | 824 Bytes | N/A |
| └─Selection_9 | 224.70 | 4249330.10 | 3 | cop[tikv] | | tikv_task:{time:9ms, loops:7}, scan_detail: {total_process_keys: 3000, total_process_keys_size: 1929174, total_keys: 3001, get_snapshot_time: 10.5µs, rocksdb: {key_skipped_count: 5999, block: {cache_hit_count: 2, read_count: 33, read_byte: 1.23 MB, read_time: 318.7µs}}} | eq(tpcc.customer.c_last, "ANTIOUGHTPRI") | N/A | N/A |
| └─TableRangeScan_8 | 9531.70 | 3773698.41 | 3000 | cop[tikv] | table:customer | tikv_task:{time:9ms, loops:7} | range:[336 7,336 7], keep order:false | N/A | N/A |
```
Good plan:
```
| id | estRows | estCost | actRows | task | access object | execution info | operator info | memory | disk |
| Projection_14 | 171.09 | 338748.36 | 1 | root | | time:1.36ms, loops:2, Concurrency:OFF | tpcc.customer.c_balance, tpcc.customer.c_first, tpcc.customer.c_middle, tpcc.customer.c_id | 5.24 KB | N/A |
| └─IndexLookUp_17 | 171.09 | 338680.06 | 1 | root | | time:1.35ms, loops:2, index_task: {total_time: 502.1µs, fetch_handle: 498.8µs, build: 2.07µs, wait: 1.17µs}, table_task: {total_time: 782.1µs, num: 1, concurrency: 5}, next: {wait_index: 592.7µs, wait_table_lookup_build: 12.3µs, wait_table_lookup_resp: 731.3µs} | | 21.9 KB | N/A |
| ├─IndexRangeScan_15(Build) | 171.09 | 47633.00 | 1 | cop[tikv] | table:customer, index:idx_customer(c_w_id, c_d_id, c_last, c_first) | time:490µs, loops:3, cop_task: {num: 1, max: 450.9µs, proc_keys: 1, tot_proc: 62.1µs, tot_wait: 42.4µs, rpc_num: 1, rpc_time: 436.1µs, copr_cache_hit_ratio: 0.00, build_task_duration: 13.4µs, max_distsql_concurrency: 1}, tikv_task:{time:0s, loops:1}, scan_detail: {total_process_keys: 1, total_process_keys_size: 118, total_keys: 2, get_snapshot_time: 8.7µs, rocksdb: {key_skipped_count: 1, block: {cache_hit_count: 8}}} | range:[582 10 "OUGHTPRICALLY",582 10 "OUGHTPRICALLY"], keep order:true | N/A | N/A |
| └─TableRowIDScan_16(Probe) | 171.09 | 67744.96 | 1 | cop[tikv] | table:customer | time:691.5µs, loops:2, cop_task: {num: 1, max: 631.6µs, proc_keys: 1, tot_proc: 221.5µs, tot_wait: 53.2µs, rpc_num: 1, rpc_time: 616.2µs, copr_cache_hit_ratio: 0.00, build_task_duration: 22µs, max_distsql_concurrency: 1, max_extra_concurrency: 1}, tikv_task:{time:0s, loops:1}, scan_detail: {total_process_keys: 1, total_process_keys_size: 565, total_keys: 2, get_snapshot_time: 8.81µs, rocksdb: {key_skipped_count: 1, block: {cache_hit_count: 14, read_count: 3, read_byte: 75.1 KB, read_time: 32.6µs}}} | keep order:false | N/A | N/A |
```
For this query, it's more likely to pick slower plan:

# Second Query
```
SELECT `o_id`, `o_carrier_id`, `o_entry_d` FROM `orders` WHERE `o_w_id` = ? AND `o_d_id` = ? AND `o_c_id` = ? ORDER BY `o_id` DESC LIMIT ?
```
Wrong plan:
```
| id | estRows | estCost | actRows | task | access object | execution info | operator info | memory | disk |
| Projection_7 | 1.00 | 1168.13 | 1 | root | | time:3.66ms, loops:2, Concurrency:OFF | tpcc.orders.o_id, tpcc.orders.o_carrier_id, tpcc.orders.o_entry_d | 49.5 KB | N/A |
| └─Limit_11 | 1.00 | 1167.83 | 1 | root | | time:3.65ms, loops:2 | offset:0, count:1 | N/A | N/A |
| └─TableReader_24 | 1.00 | 1167.83 | 1 | root | | time:3.65ms, loops:1, cop_task: {num: 1, max: 3.6ms, proc_keys: 2016, tot_proc: 3ms, tot_wait: 42.9µs, rpc_num: 1, rpc_time: 3.54ms, copr_cache_hit_ratio: 0.00, build_task_duration: 5.07µs, max_distsql_concurrency: 1} | data:Limit_23 | 409 Bytes | N/A |
| └─Limit_23 | 1.00 | 17327.32 | 1 | cop[tikv] | | tikv_task:{time:3ms, loops:6}, scan_detail: {total_process_keys: 2016, total_process_keys_size: 155577, total_keys: 2017, get_snapshot_time: 10.6µs, rocksdb: {key_skipped_count: 2017, block: {cache_hit_count: 3, read_count: 3, read_byte: 18.7 KB, read_time: 18.3µs}}} | offset:0, count:1 | N/A | N/A |
| └─Selection_22 | 1.00 | 17327.32 | 1 | cop[tikv] | | tikv_task:{time:3ms, loops:6} | eq(tpcc.orders.o_c_id, 1464) | N/A | N/A |
| └─TableRangeScan_21 | 36.74 | 15493.79 | 2016 | cop[tikv] | table:orders | tikv_task:{time:3ms, loops:6} | range:[211 7,211 7], keep order:true, desc | N/A | N/A |
```
Good plan:
```
| id | estRows | estCost | actRows | task | access object | execution info | operator info | memory | disk |
| Projection_7 | 1.00 | 1187.42 | 1 | root | | time:2.27ms, loops:2, Concurrency:OFF | tpcc.orders.o_id, tpcc.orders.o_carrier_id, tpcc.orders.o_entry_d | 17.7 KB | N/A |
| └─IndexLookUp_28 | 1.00 | 1187.12 | 1 | root | | time:2.27ms, loops:2, index_task: {total_time: 829.2µs, fetch_handle: 821.8µs, build: 2.35µs, wait: 5.11µs}, table_task: {total_time: 951.6µs, num: 1, concurrency: 5}, next: {wait_index: 1.3ms, wait_table_lookup_build: 318µs, wait_table_lookup_resp: 631.1µs} | limit embedded(offset:0, count:1) | 19.4 KB | N/A |
| ├─Limit_27(Build) | 1.00 | 415.59 | 1 | cop[tikv] | | time:812.4µs, loops:1, cop_task: {num: 1, max: 777.8µs, proc_keys: 1, tot_proc: 198.8µs, tot_wait: 48.1µs, rpc_num: 1, rpc_time: 748.4µs, copr_cache_hit_ratio: 0.00, build_task_duration: 430.5µs, max_distsql_concurrency: 1}, tikv_task:{time:0s, loops:1}, scan_detail: {total_process_keys: 1, total_process_keys_size: 100, total_keys: 2, get_snapshot_time: 11.4µs, rocksdb: {key_skipped_count: 1, block: {cache_hit_count: 2, read_count: 2, read_byte: 9.69 KB, read_time: 20µs}}} | offset:0, count:1 | N/A | N/A |
| │ └─IndexRangeScan_25 | 1.00 | 415.59 | 1 | cop[tikv] | table:orders, index:idx_order(o_w_id, o_d_id, o_c_id, o_id) | tikv_task:{time:0s, loops:1} | range:[699 3 740,699 3 740], keep order:true, desc, stats:partial[primary:allEvicted, idx_order:allEvicted, o_w_id:allEvicted...(more: 1 allEvicted)] | N/A | N/A |
| └─TableRowIDScan_26(Probe) | 1.00 | 244.20 | 1 | cop[tikv] | table:orders | time:605.9µs, loops:2, cop_task: {num: 1, max: 560.6µs, proc_keys: 1, tot_proc: 155.7µs, tot_wait: 18.7µs, rpc_num: 1, rpc_time: 534.9µs, copr_cache_hit_ratio: 0.00, build_task_duration: 293.5µs, max_distsql_concurrency: 1, max_extra_concurrency: 1}, tikv_task:{time:0s, loops:1}, scan_detail: {total_process_keys: 1, total_process_keys_size: 79, total_keys: 2, get_snapshot_time: 9.31µs, rocksdb: {key_skipped_count: 1, block: {cache_hit_count: 1, read_count: 1, read_byte: 6.16 KB, read_time: 8.03µs}}} | keep order:false, stats:partial[primary:allEvicted, idx_order:allEvicted, o_w_id:allEvicted...(more: 1 allEvicted)] | N/A | N/A |
```
For this query, it's more likely to pick faster plan, but it's also possible to pick slower plan some time:

I tried to use the following bindings as a workaround:
```
CREATE GLOBAL BINDING FOR
SELECT c_balance, c_first, c_middle, c_id FROM customer WHERE c_w_id = ? AND c_d_id = ? AND c_last = ? ORDER BY c_first
USING
SELECT /*+ use_index(customer, idx_customer) */ c_balance, c_first, c_middle, c_id FROM customer WHERE c_w_id = ? AND c_d_id = ? AND c_last = ? ORDER BY c_first;
CREATE GLOBAL BINDING FOR
SELECT `o_id`, `o_carrier_id`, `o_entry_d` FROM `orders` WHERE `o_w_id` = ? AND `o_d_id` = ? AND `o_c_id` = ? ORDER BY `o_id` DESC LIMIT ?
USING
SELECT /*+ use_index(orders, idx_order) */ `o_id`, `o_carrier_id`, `o_entry_d` FROM `orders` WHERE `o_w_id` = ? AND `o_d_id` = ? AND `o_c_id` = ? ORDER BY `o_id` DESC LIMIT ?;
```
If you need any help to reproduce this issue, feel free to touch me.
Contributor guide
Assessment
This issue has not been assessed yet.