tikv / tikv/pd

There is a 989.3% performance regression in pxxxxxxr after PR#10621

Open
#10,830 5 comments 0 reactions 1 assignee Claimed by @lhy1024 View on GitHub
affects-8.5 contribution severity/critical type/bug type/regression
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?

1. Deploy a cluster with 3 TiDB and 3 Tikv.
2. Run benchmark pxxxxxxr.

### What did you expect to see?

No performance regression.

### What did you see instead?

TikvCoprocessorAvg regression in benchmark pxxxxxxr compared to baseline 137.11 after #10621.

| description | tidb | tikv | pd | TikvCoprocessorAvg avg |
| ----------------- | ------------------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------- |
| last success | pingcap/tidb@84548dbcc1739038be43f4d18de595dc371cea2e | tikv/tikv@b96b66319b5663e2c04e7ed3e9f167bf61ce4ac7 | tikv/pd@7a78ca8644b6adac5f6cb62ccfef897e3cb95156 | 84.67, 109.32 |
| first regression | pingcap/tidb@84548dbcc1739038be43f4d18de595dc371cea2e | tikv/tikv@b96b66319b5663e2c04e7ed3e9f167bf61ce4ac7 | tikv/pd@38a0f9ab0a3e373c2da23fa7c556423b4a67a6db | 1493.55, 1362.70 |

### What version of PD are you using?

tikv/pd@38a0f9ab0a3e373c2da23fa7c556423b4a67a6db

### Root Cause Analysis

PD commit 38a0f9ab (tikv/pd#10621) adds load-based split-scatter. When TiKV reports AskBatchSplit with SplitReason_LOAD, RecordSplitScatterBatch records the batch, and dispatchSplitScatterRegions creates scatter operators on every patrol cycle, causing PD IO doubling, TiKV coprocessor explosion, and extreme tail latency.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.