pingcap / pingcap/tidb

[lightning]: Do we have some rate limit mechanism on split/scatter regions for lightning?

Open
#38,492 0 comments 0 reactions 1 assignee Claimed by @niubell View on GitHub
component/lightning type/question
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## General Question

To my acknowledge, we only have the param 'store_write_bwlimit' to limit the import rate for lightning.

But one user of br have made some tests which limit the number of 'scatter regions' and show the huge improvement on impact for the online application.
```
When I tested the lightning(6.2) local backend import, I noticed that when lightning starts to import data to the cluster, it will cause a spike on both read/write latency increase. For example, insert p95 from 15ms to 172ms, read p95 from 2ms to 95ms. It only lasts for 2 mins.

At first I thought it’s because lightning is high. But when I limit the import speed with store-write-bwlimit = "10MiB", I noticed the import cpu drops from 30% to 6%, but still see the latency spike

then I suspect that it’s because at the beginning of import, it creates a lot of scatter region operators and causes balance leader.

I did some tests, if you don’t change anything, lightning will scatter and split [4096 keys](https://github.com/pingcap/tidb/blob/master/br/pkg/lightning/backend/local/localhelper.go#L53), which will cause the spike above
- 5k ops/m scatter-region
- insert p95 from 15ms to 172ms, p99 from 28ms to 790ms
- read p95 from 2ms to 95ms, p99 from 2.5ms to 713ms

I changed the source code of lightning to split max 1000 keys at a time and between each split it sleeps 10 mins. You can see about
- 1.3k ops/m scatter-region every 10 mins
- insert p95 from 15ms to 26ms, p99 28ms to 156ms
- read p95 not changed, p99 from 2.5ms to 173 ms
```

Can we do some inner tests to confirm the impact of 'scatter region' for lightning?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.