[lightning]: Do we have some rate limit mechanism on split/scatter regions for lightning?
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## General Question
To my acknowledge, we only have the param 'store_write_bwlimit' to limit the import rate for lightning.
But one user of br have made some tests which limit the number of 'scatter regions' and show the huge improvement on impact for the online application.
```
When I tested the lightning(6.2) local backend import, I noticed that when lightning starts to import data to the cluster, it will cause a spike on both read/write latency increase. For example, insert p95 from 15ms to 172ms, read p95 from 2ms to 95ms. It only lasts for 2 mins.
At first I thought it’s because lightning is high. But when I limit the import speed with store-write-bwlimit = "10MiB", I noticed the import cpu drops from 30% to 6%, but still see the latency spike
then I suspect that it’s because at the beginning of import, it creates a lot of scatter region operators and causes balance leader.
I did some tests, if you don’t change anything, lightning will scatter and split [4096 keys](https://github.com/pingcap/tidb/blob/master/br/pkg/lightning/backend/local/localhelper.go#L53), which will cause the spike above
- 5k ops/m scatter-region
- insert p95 from 15ms to 172ms, p99 from 28ms to 790ms
- read p95 from 2ms to 95ms, p99 from 2.5ms to 713ms
I changed the source code of lightning to split max 1000 keys at a time and between each split it sleeps 10 mins. You can see about
- 1.3k ops/m scatter-region every 10 mins
- insert p95 from 15ms to 26ms, p99 28ms to 156ms
- read p95 not changed, p99 from 2.5ms to 173 ms
```
Can we do some inner tests to confirm the impact of 'scatter region' for lightning?
Contributor guide
Assessment
This issue has not been assessed yet.