High query error rate when a new member becomes leader without region metadata
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
### What did you do?
Added a new node to an existing 5-node PD cluster and restarted the existing leader as soon as new PD member got added. Note that both were coordinated and automated cluster maintenance operations.
### What did you expect to see?
New PD member sync region history from an existing leader, and cluster continue to operate with a healthy PD leader.
### What did you see instead?
The new PD member becomes the leader without a chance to recover region metadata. The empty region metadata cache caused region heartbeats to contend on the region cache/b-tree, increasing both region heartbeat and GetRegion latencies, eventually affecting query latencies, and increasing error rates.
Failed Query OPM
99% Region heartbeat handle latency (from PD)
Load Region Duration (from TiDB)
### What version of PD are you using (`pd-server -V`)?
v8.5.2
Contributor guide
Assessment
This issue has not been assessed yet.