tikv / tikv/pd

Bootstrap failed continuously when disk slow

Open
#6,311 0 comments 0 reactions 0 assignees View on GitHub
good first issue type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?
Deploy a TiDB cluster.

### What did you expect to see?
tikv can bootstrap successfully even after many retries

### What did you see instead?
PD
``` log
[2023/04/11 19:46:57.090 +08:00] [INFO] [server.go:631] ["try to bootstrap raft cluster"] [cluster-id=7220757462908539357] [request="header: store: region: peers: > "]
[2023/04/11 19:46:59.094 +08:00] [INFO] [server.go:631] ["try to bootstrap raft cluster"] [cluster-id=7220757462908539357] [request="header: store: region: peers: > "]
[2023/04/11 19:47:01.097 +08:00] [INFO] [server.go:631] ["try to bootstrap raft cluster"] [cluster-id=7220757462908539357] [request="header: store: region: peers: > "]
[2023/04/11 19:47:03.101 +08:00] [INFO] [server.go:631] ["try to bootstrap raft cluster"] [cluster-id=7220757462908539357] [request="header: store: region: peers: > "]
[2023/04/11 19:47:07.091 +08:00] [WARN] [etcd_kv.go:160] ["txn runs too slow"] [response=null] [cost=10.000348001s] [error="context deadline exceeded"]
[2023/04/11 19:47:07.091 +08:00] [WARN] [server.go:685] ["bootstrap failed to update etcd"] [error="context deadline exceeded"]
[2023/04/11 19:47:09.094 +08:00] [WARN] [etcd_kv.go:160] ["txn runs too slow"] [response=null] [cost=10.000328776s] [error="context deadline exceeded"]
[2023/04/11 19:47:09.094 +08:00] [WARN] [server.go:685] ["bootstrap failed to update etcd"] [error="context deadline exceeded"]
[2023/04/11 19:47:10.657 +08:00] [WARN] [server.go:689] ["cluster already bootstrapped"] [cluster-id=7220757462908539357]
[2023/04/11 19:47:10.657 +08:00] [WARN] [server.go:689] ["cluster already bootstrapped"] [cluster-id=7220757462908539357]
```
After several retries, an error was reported in TiKV.
```
[2023/04/11 19:47:15.127 +08:00] [WARN] [node.rs:469] ["get the first region failed"] [err="RegionNotFound([])"]
[2023/04/11 19:47:18.129 +08:00] [WARN] [node.rs:469] ["get the first region failed"] [err="RegionNotFound([])"]
[2023/04/11 19:47:21.131 +08:00] [WARN] [node.rs:469] ["get the first region failed"] [err="RegionNotFound([])"]
```

### What version of PD are you using (`pd-server -V`)?
v6.5.1

gRPC context deadline exceeded, but etcd still finished the txn. And PD don't save region into LevelDB.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.