tikv / tikv/pd

qps drop to zero and also influence the lag of pitr or cdc during injection pd leader io hang or io delay 500ms or 1s due to transfer etcd leader failed

Open
#8,204 4 comments 0 reactions 1 assignee Claimed by @JmPotato View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?
1、run tpcc
2、inject pd leader io hang last for 5m

### What did you expect to see?
qps can recover within 2mins

### What did you see instead?
qps drop to zero during injection pd leader io hang
![img_v3_02b3_fcc533fe-6dec-40c5-8a12-cc9601a3434g](https://github.com/tikv/pd/assets/84712107/6344587b-6152-45f7-947c-30314e574e9c)

![img_v3_02b3_ed19d944-62f5-4cf9-95b4-412749df9c6g](https://github.com/tikv/pd/assets/84712107/04937d90-0735-4944-b5ca-9bdfd209b4cc)

io delay 500ms
![image](https://github.com/tikv/pd/assets/84712107/6e293f9b-ee99-4e7d-9492-1526c2668c41)

![image](https://github.com/tikv/pd/assets/84712107/aebda1ab-dbe4-440e-b6d4-c7dd207da765)

### What version of PD are you using (`pd-server -V`)?
./pd-server -V
Release Version: v8.1.0
Edition: Community
Git Commit Hash: fca469ca33eb5d8b5e0891b507c87709a00b0e81
Git Branch: HEAD
UTC Build Time: 2024-05-09 02:15:45
2024-05-17T01:42:48.440+0800

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.