tikv / tikv/pd

PD should evict all leaders from the deleted store before moving replicas aggressively

Open
#4,099 7 comments 0 reactions 1 assignee Claimed by @nolouch View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?
Delete a store with the default settings.

### What did you expect to see?
The cluster continue to operate smoothly with slightly increasing latency.

### What did you see instead?
The cluster has a lot of jitters.

When deleting a store, PD will set remove peer limit to 100000000, which means the speed depends on the number of available up stores. When deleting a lot of ranges in TiKV, it will change the LSM tree rapidly and causing a lot of compaction, which will consume both CPU and IO. If there are still leaders on the deleted store, the latency can be very high and unstable.

![image](https://user-images.githubusercontent.com/1701473/132510303-e4c31e72-16ba-4021-84cf-6ee693fd721a.png)

I added an evict leader scheduler manually in the left part, and remove it in the right part. The latency and throughput is changed dramatically.

![image](https://user-images.githubusercontent.com/1701473/132510253-dcb1bd32-b2e8-4786-8c0f-4a4f5d9098c9.png)

### What version of PD are you using (`pd-server -V`)?
```
Release Version: v5.1.1
Edition: Community
Git Commit Hash: 7cba1912b317a533e18b16ea2ba9a14ed2891129
Git Branch: heads/refs/tags/v5.1.1
UTC Build Time: 2021-07-23 01:16:18
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.