replace down-peer when peer is down while the store is still connected
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
Down-peer is detected and reported by TiKV through heartbeat to PD. However, when PD checks the placement-rule and finds there is a down-peer in one region, it will first check if the peer's tikv is down, if not, it will take no action(skip directly without any log).
https://github.com/tikv/pd/blob/da5a4e95ca0a739fc91e4dfd0f281e5413f5ff15/pkg/schedule/checker/rule_checker.go#L190-L203
However, in some cases, when there are issues with the internal region raft group of TiKV, it may cause some replicas to fail to maintain raft heartbeats and result in down-peers, while TiKV can still report heartbeats to PD normally. In this situation, down-peers will continue to exist, resulting in incomplete replica numbers for a long period of time for some regions.
I think in this situation, if a peer has been without heartbeat for a certain period of time(down-peer), regardless of whether tikv is in a down state or not, we should try to recover these down-peers on the PD scheduling side just like replace-offline-peers.
Contributor guide
Assessment
This issue has not been assessed yet.