TiDB can't serve `tidb_replica_read='closest-replicas'` for some time after one of TiKV restart if there were a region split during restart
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
In a cluster with zone labels:
1. Set TiKV configs to split regions agressivly:
```
[coprocessor]
region-max-keys = 64000
region-split-keys = 51200
```
2. Run the load generating a lot of new regions:
```
sysbench oltp_read_only --skip_trx=true --create_secondary=false --tables=8 --table_size=20000000 --threads=8 --mysql-host=${HOST} --mysql-user=root --mysql-db=test --mysql-port=4000 prepare
```
3. Run read traffic with `SESSION="SET max_execution_time=1100;SET tidb_replica_read='closest-replicas'"`
```
sysbench oltp_read_only --mysql-ignore-errors=1213,1020,1205,1105,9005 --point_selects=0 --range_selects=true --range_size=2 --simple_ranges=1 --sum_ranges=0 --order_ranges=0 --distinct_ranges=0 --distinct_ranges=0 --skip_trx=true --tables=2 --table_size=20000000 --threads=2 --mysql-host=${HOST} --mysql-user=root --mysql-db=test --mysql-port=4000 --report-interval=1 --time=900 run
```
4. Restart TiKV node with the same label as TiDB with 2 minute downtime and wait till it comes back.
### 2. What did you expect to see? (Required)
TiDB traffic should not see any significant impact after the node comes back.
### 3. What did you see instead (Required)
The QPS drops almost to zero
```
[ 25s ] thds: 2 tps: 1328.98 qps: 1328.98 (r/w/o: 1328.98/0.00/0.00) lat (ms,95%): 1.73 err/s: 0.00 reconn/s: 0.00
[ 26s ] thds: 2 tps: 1337.88 qps: 1337.88 (r/w/o: 1337.88/0.00/0.00) lat (ms,95%): 1.76 err/s: 0.00 reconn/s: 0.00
[ 27s ] thds: 2 tps: 1348.08 qps: 1348.08 (r/w/o: 1348.08/0.00/0.00) lat (ms,95%): 1.73 err/s: 0.00 reconn/s: 0.00
[ 28s ] thds: 2 tps: 1393.97 qps: 1393.97 (r/w/o: 1393.97/0.00/0.00) lat (ms,95%): 1.70 err/s: 0.00 reconn/s: 0.00
[ 29s ] thds: 2 tps: 1364.95 qps: 1364.95 (r/w/o: 1364.95/0.00/0.00) lat (ms,95%): 1.70 err/s: 0.00 reconn/s: 0.00
# tikv in the same zone goes down, traffic auto rerouted to other AZ with higher network latency so the sysbench traffic drops
[ 30s ] thds: 2 tps: 994.07 qps: 994.07 (r/w/o: 994.07/0.00/0.00) lat (ms,95%): 2.57 err/s: 0.00 reconn/s: 0.00
[ 31s ] thds: 2 tps: 854.94 qps: 854.94 (r/w/o: 854.94/0.00/0.00) lat (ms,95%): 2.81 err/s: 0.00 reconn/s: 0.00
[ 32s ] thds: 2 tps: 872.05 qps: 872.05 (r/w/o: 872.05/0.00/0.00) lat (ms,95%): 2.71 err/s: 0.00 reconn/s: 0.00
[ 33s ] thds: 2 tps: 854.27 qps: 854.27 (r/w/o: 854.27/0.00/0.00) lat (ms,95%): 2.81 err/s: 0.00 reconn/s: 0.00
...
[ 182s ] thds: 2 tps: 872.60 qps: 872.60 (r/w/o: 872.60/0.00/0.00) lat (ms,95%): 2.61 err/s: 0.00 reconn/s: 0.00
# tikv comes back and the traffic is completely stalled
[ 183s ] thds: 2 tps: 430.20 qps: 430.20 (r/w/o: 430.20/0.00/0.00) lat (ms,95%): 5.00 err/s: 0.00 reconn/s: 0.00
[ 184s ] thds: 2 tps: 8.00 qps: 8.00 (r/w/o: 8.00/0.00/0.00) lat (ms,95%): 1129.24 err/s: 2.00 reconn/s: 0.00
[ 185s ] thds: 2 tps: 12.00 qps: 12.00 (r/w/o: 12.00/0.00/0.00) lat (ms,95%): 5.67 err/s: 2.00 reconn/s: 0.00
[ 186s ] thds: 2 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 187s ] thds: 2 tps: 10.00 qps: 10.00 (r/w/o: 10.00/0.00/0.00) lat (ms,95%): 2405.65 err/s: 2.00 reconn/s: 0.00
...
[ 438s ] thds: 2 tps: 5.00 qps: 5.00 (r/w/o: 5.00/0.00/0.00) lat (ms,95%): 2405.65 err/s: 2.00 reconn/s: 0.00
[ 439s ] thds: 2 tps: 4.00 qps: 4.00 (r/w/o: 4.00/0.00/0.00) lat (ms,95%): 1191.92 err/s: 2.00 reconn/s: 0.00
# restarted tikv is caughtup
[ 440s ] thds: 2 tps: 583.25 qps: 583.25 (r/w/o: 583.25/0.00/0.00) lat (ms,95%): 5.09 err/s: 0.00 reconn/s: 0.00
[ 441s ] thds: 2 tps: 920.01 qps: 920.01 (r/w/o: 920.01/0.00/0.00) lat (ms,95%): 5.00 err/s: 0.00 reconn/s: 0.00
[ 442s ] thds: 2 tps: 954.96 qps: 954.96 (r/w/o: 954.96/0.00/0.00) lat (ms,95%): 5.00 err/s: 0.00 reconn/s: 0.00
````
### 4. What is your TiDB version? (Required)
8.5.2
Here is tidb log from similar experiment but default max_execution_time=40 seconds
```
[2025/08/09 00:12:38.181 +00:00] [INFO] [region_request.go:976] ["send request meet region error without retry"] [conn=3894411422] [session_alias=] [req-ts=459983634542362624] [req-type=Cop] [region="{ region id: 18381, ver: 1564, confVer: 5 }"] [replica-read-type=mixed] [stale-read=true] [request-sender="{rpcError:, replicaSelector: replicaSelectorV2{replicaReadType: mixed, attempts: 1, cacheRegionIsValid: false, replicaStatus: [peer: 18382, store: 1001, isEpochStale: false, attempts: 0, attempts_time: 0s, replica-epoch: 0, store-epoch: 0, store-state: resolved, store-liveness-state: reachable peer: 18383, store: 1130, isEpochStale: false, attempts: 0, attempts_time: 0s, replica-epoch: 0, store-epoch: 0, store-state: resolved, store-liveness-state: reachable peer: 18384, store: 1195, isEpochStale: false, attempts: 1, attempts_time: 476.9µs, replica-epoch: 1, store-epoch: 1, store-state: resolved, store-liveness-state: reachable]}}"] [total-round-stats="{total-backoff: 40s, total-backoff-times: 87, total-rpc: {Cop:{num_rpc:88, total_time:44.7ms}, rpc_errors:{region_not_found:88}}}"] [current-round-stats="{time: 522.2µs, backoff: 0s, timeout: 1m0s, req-max-exec-timeout: 1m0s, retry-times: 0, replica-access: {{stale_read, peer:18384, store:1195, err:region_not_found}}}"]
```
Contributor guide
Assessment
This issue has not been assessed yet.