tikv / tikv/pd

PD can't schedule well with the simple sysbench workload

Open
#4,041 2 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?
Deploy a cluster with 3 tikv, 1 pd and 1 tidb. Run sysbench prepare with 32 threads and 32 tables, each table has more than 100000000 rows.

### What did you expect to see?
The cluster runs with balanced load on all three tikv instances.

### What did you see instead?
Only one of the tikv instance in high load.

The problem is pd doesn't consider leader during scheduling. After scattering leaders manually, the throughput is increased by more than 10%.

0. Why leader count matters?
Leader is supposed to handle proposals and apply logs as fast as possible, so all logs are committed and applied quickly on leader; while followers can be slow and apply logs lazily. This means the working mode between leader and followers are different and resource consumptions are also different. If all leaders are scheduled to one node, it can lead to significant hotspot on one store.

1. Why just considering written bytes/keys is not enough?
There are only three TiKV instances and only three replicas, so the throughput should be the same at a given period no matter leaders are balanced or not.

2. Why just considering QPS is not enough?
Hotspots can cause leader performs badly. The sum QPS of several badly performed leaders can be the same as a well performed leader. So even QPS reach balance doesn't mean it can't get better.

### What version of PD are you using (`pd-server -V`)?
v5.1.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.