Bootstrap failed continuously when disk slow
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
### What did you do?
Deploy a TiDB cluster.
### What did you expect to see?
tikv can bootstrap successfully even after many retries
### What did you see instead?
PD
``` log
[2023/04/11 19:46:57.090 +08:00] [INFO] [server.go:631] ["try to bootstrap raft cluster"] [cluster-id=7220757462908539357] [request="header: store: region: peers: > "]
[2023/04/11 19:46:59.094 +08:00] [INFO] [server.go:631] ["try to bootstrap raft cluster"] [cluster-id=7220757462908539357] [request="header: store: region: peers: > "]
[2023/04/11 19:47:01.097 +08:00] [INFO] [server.go:631] ["try to bootstrap raft cluster"] [cluster-id=7220757462908539357] [request="header: store: region: peers: > "]
[2023/04/11 19:47:03.101 +08:00] [INFO] [server.go:631] ["try to bootstrap raft cluster"] [cluster-id=7220757462908539357] [request="header: store: region: peers: > "]
[2023/04/11 19:47:07.091 +08:00] [WARN] [etcd_kv.go:160] ["txn runs too slow"] [response=null] [cost=10.000348001s] [error="context deadline exceeded"]
[2023/04/11 19:47:07.091 +08:00] [WARN] [server.go:685] ["bootstrap failed to update etcd"] [error="context deadline exceeded"]
[2023/04/11 19:47:09.094 +08:00] [WARN] [etcd_kv.go:160] ["txn runs too slow"] [response=null] [cost=10.000328776s] [error="context deadline exceeded"]
[2023/04/11 19:47:09.094 +08:00] [WARN] [server.go:685] ["bootstrap failed to update etcd"] [error="context deadline exceeded"]
[2023/04/11 19:47:10.657 +08:00] [WARN] [server.go:689] ["cluster already bootstrapped"] [cluster-id=7220757462908539357]
[2023/04/11 19:47:10.657 +08:00] [WARN] [server.go:689] ["cluster already bootstrapped"] [cluster-id=7220757462908539357]
```
After several retries, an error was reported in TiKV.
```
[2023/04/11 19:47:15.127 +08:00] [WARN] [node.rs:469] ["get the first region failed"] [err="RegionNotFound([])"]
[2023/04/11 19:47:18.129 +08:00] [WARN] [node.rs:469] ["get the first region failed"] [err="RegionNotFound([])"]
[2023/04/11 19:47:21.131 +08:00] [WARN] [node.rs:469] ["get the first region failed"] [err="RegionNotFound([])"]
```
### What version of PD are you using (`pd-server -V`)?
v6.5.1
gRPC context deadline exceeded, but etcd still finished the txn. And PD don't save region into LevelDB.
Contributor guide
Assessment
This issue has not been assessed yet.