pingcap / pingcap/tidb

pd is causing tikv:8252 error when resource usage hasn't reach limit and io latency increased

Open
#56,644 0 comments 0 reactions 0 assignees View on GitHub
affects-7.1 affects-7.5 component/pd severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step (Required)

Production system running normal

### 2. What did you expect to see? (Required)

No spike of any issue

### 3. What did you see instead (Required)

At problem time, we see a spike of `tikv:8252 Resource Group Quota Limitation` error:
![20241014-215706](https://github.com/user-attachments/assets/e2482751-7a71-445c-8057-ef8a33bf3c56)

But the resource group has not exceed resouce limit:
![20241014-215918](https://github.com/user-attachments/assets/a4f799b0-5a2f-44f6-92d8-7ee67d35638f)

There is issue with PD:
![20241014-220037](https://github.com/user-attachments/assets/bbed89dc-b1b8-4c53-bdf2-d7bf411b2945)

And it appears there is a spike on IO latency:
![image](https://github.com/user-attachments/assets/8fde3d3f-16d4-4d77-868c-f64fdff0b183)

### 4. What is your TiDB version? (Required)
v7.5.3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.