tikv / tikv/pd

High query error rate when a new member becomes leader without region metadata

Open
#10,843 1 comment 0 reactions 0 assignees View on GitHub
contribution first-time-contributor type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?

Added a new node to an existing 5-node PD cluster and restarted the existing leader as soon as new PD member got added. Note that both were coordinated and automated cluster maintenance operations.

### What did you expect to see?

New PD member sync region history from an existing leader, and cluster continue to operate with a healthy PD leader.

### What did you see instead?

The new PD member becomes the leader without a chance to recover region metadata. The empty region metadata cache caused region heartbeats to contend on the region cache/b-tree, increasing both region heartbeat and GetRegion latencies, eventually affecting query latencies, and increasing error rates.

Failed Query OPM

Image

99% Region heartbeat handle latency (from PD)
Image

Load Region Duration (from TiDB)
Image

### What version of PD are you using (`pd-server -V`)?
v8.5.2

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.