There is a 989.3% performance regression in pxxxxxxr after PR#10621
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
### What did you do?
1. Deploy a cluster with 3 TiDB and 3 Tikv.
2. Run benchmark pxxxxxxr.
### What did you expect to see?
No performance regression.
### What did you see instead?
TikvCoprocessorAvg regression in benchmark pxxxxxxr compared to baseline 137.11 after #10621.
| description | tidb | tikv | pd | TikvCoprocessorAvg avg |
| ----------------- | ------------------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------- |
| last success | pingcap/tidb@84548dbcc1739038be43f4d18de595dc371cea2e | tikv/tikv@b96b66319b5663e2c04e7ed3e9f167bf61ce4ac7 | tikv/pd@7a78ca8644b6adac5f6cb62ccfef897e3cb95156 | 84.67, 109.32 |
| first regression | pingcap/tidb@84548dbcc1739038be43f4d18de595dc371cea2e | tikv/tikv@b96b66319b5663e2c04e7ed3e9f167bf61ce4ac7 | tikv/pd@38a0f9ab0a3e373c2da23fa7c556423b4a67a6db | 1493.55, 1362.70 |
### What version of PD are you using?
tikv/pd@38a0f9ab0a3e373c2da23fa7c556423b4a67a6db
### Root Cause Analysis
PD commit 38a0f9ab (tikv/pd#10621) adds load-based split-scatter. When TiKV reports AskBatchSplit with SplitReason_LOAD, RecordSplitScatterBatch records the batch, and dispatchSplitScatterRegions creates scatter operators on every patrol cycle, causing PD IO doubling, TiKV coprocessor explosion, and extreme tail latency.
Contributor guide
Assessment
This issue has not been assessed yet.