pingcap / pingcap/tidb

The plan of TPC-C query is not stable

Open
#52,678 5 comments 0 reactions 0 assignees View on GitHub
affects-6.5 affects-7.1 affects-7.5 affects-8.1 severity/moderate sig/planner type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

The plan of two queries in TPC-C `ORDER_STATUS` transaction is not stable:

# First Query

```
SELECT c_balance, c_first, c_middle, c_id FROM customer WHERE c_w_id = ? AND c_d_id = ? AND c_last = ? ORDER BY c_first
```

Wrong plan:

```
| id | estRows | estCost | actRows | task | access object | execution info | operator info | memory | disk |
| Sort_5 | 224.70 | 382240.44 | 3 | root | | time:9.41ms, loops:2 | tpcc.customer.c_first | 1.89 KB | 0 Bytes |
| └─Projection_7 | 224.70 | 291683.87 | 3 | root | | time:9.37ms, loops:2, Concurrency:OFF | tpcc.customer.c_balance, tpcc.customer.c_first, tpcc.customer.c_middle, tpcc.customer.c_id | 3.21 KB | N/A |
| └─TableReader_10 | 224.70 | 291594.17 | 3 | root | | time:9.37ms, loops:2, cop_task: {num: 1, max: 9.32ms, proc_keys: 3000, tot_proc: 8.93ms, tot_wait: 35.8µs, rpc_num: 1, rpc_time: 9.29ms, copr_cache_hit_ratio: 0.00, build_task_duration: 4.8µs, max_distsql_concurrency: 1} | data:Selection_9 | 824 Bytes | N/A |
| └─Selection_9 | 224.70 | 4249330.10 | 3 | cop[tikv] | | tikv_task:{time:9ms, loops:7}, scan_detail: {total_process_keys: 3000, total_process_keys_size: 1929174, total_keys: 3001, get_snapshot_time: 10.5µs, rocksdb: {key_skipped_count: 5999, block: {cache_hit_count: 2, read_count: 33, read_byte: 1.23 MB, read_time: 318.7µs}}} | eq(tpcc.customer.c_last, "ANTIOUGHTPRI") | N/A | N/A |
| └─TableRangeScan_8 | 9531.70 | 3773698.41 | 3000 | cop[tikv] | table:customer | tikv_task:{time:9ms, loops:7} | range:[336 7,336 7], keep order:false | N/A | N/A |
```

Good plan:

```

| id | estRows | estCost | actRows | task | access object | execution info | operator info | memory | disk |
| Projection_14 | 171.09 | 338748.36 | 1 | root | | time:1.36ms, loops:2, Concurrency:OFF | tpcc.customer.c_balance, tpcc.customer.c_first, tpcc.customer.c_middle, tpcc.customer.c_id | 5.24 KB | N/A |
| └─IndexLookUp_17 | 171.09 | 338680.06 | 1 | root | | time:1.35ms, loops:2, index_task: {total_time: 502.1µs, fetch_handle: 498.8µs, build: 2.07µs, wait: 1.17µs}, table_task: {total_time: 782.1µs, num: 1, concurrency: 5}, next: {wait_index: 592.7µs, wait_table_lookup_build: 12.3µs, wait_table_lookup_resp: 731.3µs} | | 21.9 KB | N/A |
| ├─IndexRangeScan_15(Build) | 171.09 | 47633.00 | 1 | cop[tikv] | table:customer, index:idx_customer(c_w_id, c_d_id, c_last, c_first) | time:490µs, loops:3, cop_task: {num: 1, max: 450.9µs, proc_keys: 1, tot_proc: 62.1µs, tot_wait: 42.4µs, rpc_num: 1, rpc_time: 436.1µs, copr_cache_hit_ratio: 0.00, build_task_duration: 13.4µs, max_distsql_concurrency: 1}, tikv_task:{time:0s, loops:1}, scan_detail: {total_process_keys: 1, total_process_keys_size: 118, total_keys: 2, get_snapshot_time: 8.7µs, rocksdb: {key_skipped_count: 1, block: {cache_hit_count: 8}}} | range:[582 10 "OUGHTPRICALLY",582 10 "OUGHTPRICALLY"], keep order:true | N/A | N/A |
| └─TableRowIDScan_16(Probe) | 171.09 | 67744.96 | 1 | cop[tikv] | table:customer | time:691.5µs, loops:2, cop_task: {num: 1, max: 631.6µs, proc_keys: 1, tot_proc: 221.5µs, tot_wait: 53.2µs, rpc_num: 1, rpc_time: 616.2µs, copr_cache_hit_ratio: 0.00, build_task_duration: 22µs, max_distsql_concurrency: 1, max_extra_concurrency: 1}, tikv_task:{time:0s, loops:1}, scan_detail: {total_process_keys: 1, total_process_keys_size: 565, total_keys: 2, get_snapshot_time: 8.81µs, rocksdb: {key_skipped_count: 1, block: {cache_hit_count: 14, read_count: 3, read_byte: 75.1 KB, read_time: 32.6µs}}} | keep order:false | N/A | N/A |
```

For this query, it's more likely to pick slower plan:

![image](https://github.com/pingcap/tidb/assets/5244316/8019e79c-d288-4e9d-bd56-d392fd89d26a)

# Second Query

```
SELECT `o_id`, `o_carrier_id`, `o_entry_d` FROM `orders` WHERE `o_w_id` = ? AND `o_d_id` = ? AND `o_c_id` = ? ORDER BY `o_id` DESC LIMIT ?
```

Wrong plan:

```

| id | estRows | estCost | actRows | task | access object | execution info | operator info | memory | disk |
| Projection_7 | 1.00 | 1168.13 | 1 | root | | time:3.66ms, loops:2, Concurrency:OFF | tpcc.orders.o_id, tpcc.orders.o_carrier_id, tpcc.orders.o_entry_d | 49.5 KB | N/A |
| └─Limit_11 | 1.00 | 1167.83 | 1 | root | | time:3.65ms, loops:2 | offset:0, count:1 | N/A | N/A |
| └─TableReader_24 | 1.00 | 1167.83 | 1 | root | | time:3.65ms, loops:1, cop_task: {num: 1, max: 3.6ms, proc_keys: 2016, tot_proc: 3ms, tot_wait: 42.9µs, rpc_num: 1, rpc_time: 3.54ms, copr_cache_hit_ratio: 0.00, build_task_duration: 5.07µs, max_distsql_concurrency: 1} | data:Limit_23 | 409 Bytes | N/A |
| └─Limit_23 | 1.00 | 17327.32 | 1 | cop[tikv] | | tikv_task:{time:3ms, loops:6}, scan_detail: {total_process_keys: 2016, total_process_keys_size: 155577, total_keys: 2017, get_snapshot_time: 10.6µs, rocksdb: {key_skipped_count: 2017, block: {cache_hit_count: 3, read_count: 3, read_byte: 18.7 KB, read_time: 18.3µs}}} | offset:0, count:1 | N/A | N/A |
| └─Selection_22 | 1.00 | 17327.32 | 1 | cop[tikv] | | tikv_task:{time:3ms, loops:6} | eq(tpcc.orders.o_c_id, 1464) | N/A | N/A |
| └─TableRangeScan_21 | 36.74 | 15493.79 | 2016 | cop[tikv] | table:orders | tikv_task:{time:3ms, loops:6} | range:[211 7,211 7], keep order:true, desc | N/A | N/A |
```

Good plan:

```

| id | estRows | estCost | actRows | task | access object | execution info | operator info | memory | disk |
| Projection_7 | 1.00 | 1187.42 | 1 | root | | time:2.27ms, loops:2, Concurrency:OFF | tpcc.orders.o_id, tpcc.orders.o_carrier_id, tpcc.orders.o_entry_d | 17.7 KB | N/A |
| └─IndexLookUp_28 | 1.00 | 1187.12 | 1 | root | | time:2.27ms, loops:2, index_task: {total_time: 829.2µs, fetch_handle: 821.8µs, build: 2.35µs, wait: 5.11µs}, table_task: {total_time: 951.6µs, num: 1, concurrency: 5}, next: {wait_index: 1.3ms, wait_table_lookup_build: 318µs, wait_table_lookup_resp: 631.1µs} | limit embedded(offset:0, count:1) | 19.4 KB | N/A |
| ├─Limit_27(Build) | 1.00 | 415.59 | 1 | cop[tikv] | | time:812.4µs, loops:1, cop_task: {num: 1, max: 777.8µs, proc_keys: 1, tot_proc: 198.8µs, tot_wait: 48.1µs, rpc_num: 1, rpc_time: 748.4µs, copr_cache_hit_ratio: 0.00, build_task_duration: 430.5µs, max_distsql_concurrency: 1}, tikv_task:{time:0s, loops:1}, scan_detail: {total_process_keys: 1, total_process_keys_size: 100, total_keys: 2, get_snapshot_time: 11.4µs, rocksdb: {key_skipped_count: 1, block: {cache_hit_count: 2, read_count: 2, read_byte: 9.69 KB, read_time: 20µs}}} | offset:0, count:1 | N/A | N/A |
| │ └─IndexRangeScan_25 | 1.00 | 415.59 | 1 | cop[tikv] | table:orders, index:idx_order(o_w_id, o_d_id, o_c_id, o_id) | tikv_task:{time:0s, loops:1} | range:[699 3 740,699 3 740], keep order:true, desc, stats:partial[primary:allEvicted, idx_order:allEvicted, o_w_id:allEvicted...(more: 1 allEvicted)] | N/A | N/A |
| └─TableRowIDScan_26(Probe) | 1.00 | 244.20 | 1 | cop[tikv] | table:orders | time:605.9µs, loops:2, cop_task: {num: 1, max: 560.6µs, proc_keys: 1, tot_proc: 155.7µs, tot_wait: 18.7µs, rpc_num: 1, rpc_time: 534.9µs, copr_cache_hit_ratio: 0.00, build_task_duration: 293.5µs, max_distsql_concurrency: 1, max_extra_concurrency: 1}, tikv_task:{time:0s, loops:1}, scan_detail: {total_process_keys: 1, total_process_keys_size: 79, total_keys: 2, get_snapshot_time: 9.31µs, rocksdb: {key_skipped_count: 1, block: {cache_hit_count: 1, read_count: 1, read_byte: 6.16 KB, read_time: 8.03µs}}} | keep order:false, stats:partial[primary:allEvicted, idx_order:allEvicted, o_w_id:allEvicted...(more: 1 allEvicted)] | N/A | N/A |
```

For this query, it's more likely to pick faster plan, but it's also possible to pick slower plan some time:

![image](https://github.com/pingcap/tidb/assets/5244316/fce7498e-49ea-40da-9aab-a4a0aea241dd)

I tried to use the following bindings as a workaround:

```
CREATE GLOBAL BINDING FOR
SELECT c_balance, c_first, c_middle, c_id FROM customer WHERE c_w_id = ? AND c_d_id = ? AND c_last = ? ORDER BY c_first
USING
SELECT /*+ use_index(customer, idx_customer) */ c_balance, c_first, c_middle, c_id FROM customer WHERE c_w_id = ? AND c_d_id = ? AND c_last = ? ORDER BY c_first;

CREATE GLOBAL BINDING FOR
SELECT `o_id`, `o_carrier_id`, `o_entry_d` FROM `orders` WHERE `o_w_id` = ? AND `o_d_id` = ? AND `o_c_id` = ? ORDER BY `o_id` DESC LIMIT ?
USING
SELECT /*+ use_index(orders, idx_order) */ `o_id`, `o_carrier_id`, `o_entry_d` FROM `orders` WHERE `o_w_id` = ? AND `o_d_id` = ? AND `o_c_id` = ? ORDER BY `o_id` DESC LIMIT ?;
```

If you need any help to reproduce this issue, feel free to touch me.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.