lightning: concurrency setting when write to tikv
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
before 7.0.0, we can use range-concurrency to control concurrency(multiple by 2 actually) when write to tikv, but after 7.0.0, the write concurrency is `table-concurrency * range-concurrency * 2` as we start `range-concurrency * 2` workers for each engine, if we enlarge `table-concurrency` as before, the tikv write concurrency might be too large and overload tikv(io), it might increase io latency -> stuck raftstore -> leader change frequently -> cause region split fail even after retry(keeps reporting `not leader` or `leader is none` error)
https://github.com/pingcap/tidb/blob/33d7a2c9840e4f07a7caa09618572744211e23e1/br/pkg/lightning/backend/local/local.go#L1420-L1424
```
error="batch split regions failed: split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader: : [BR:Restore:ErrRestoreSplitFailed]fail to split region; split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader: : [BR:Restore:ErrRestoreSplitFailed]fail to split region; split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader: : [BR:Restore:ErrRestoreSplitFailed]fail to split region; split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader: : [BR:Restore:ErrRestoreSplitFailed]fail to split region"] [errorVerbose="the following errors occurred:
- [BR:Restore:ErrRestoreSplitFailed]fail to split region
split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader:
github.com/pingcap/tidb/br/pkg/restore/split.sendSplitRegionRequest
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/client.go:370
github.com/pingcap/tidb/br/pkg/restore/split.(*pdClient).sendSplitRegionRequest
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/client.go:315
github.com/pingcap/tidb/br/pkg/restore/split.(*pdClient).BatchSplitRegionsWithOrigin
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/client.go:413
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).BatchSplitRegions
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/localhelper.go:400
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).SplitAndScatterRegionByRanges.func3
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/localhelper.go:276
golang.org/x/sync/errgroup.(*Group).Go.func1
/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650
```
Contributor guide
Assessment
This issue has not been assessed yet.