pingcap / pingcap/tidb

lightning: concurrency setting when write to tikv

Open
#51,380 1 comment 0 reactions 0 assignees View on GitHub
component/ddl component/lightning type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
before 7.0.0, we can use range-concurrency to control concurrency(multiple by 2 actually) when write to tikv, but after 7.0.0, the write concurrency is `table-concurrency * range-concurrency * 2` as we start `range-concurrency * 2` workers for each engine, if we enlarge `table-concurrency` as before, the tikv write concurrency might be too large and overload tikv(io), it might increase io latency -> stuck raftstore -> leader change frequently -> cause region split fail even after retry(keeps reporting `not leader` or `leader is none` error)

https://github.com/pingcap/tidb/blob/33d7a2c9840e4f07a7caa09618572744211e23e1/br/pkg/lightning/backend/local/local.go#L1420-L1424

```
error="batch split regions failed: split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader: : [BR:Restore:ErrRestoreSplitFailed]fail to split region; split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader: : [BR:Restore:ErrRestoreSplitFailed]fail to split region; split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader: : [BR:Restore:ErrRestoreSplitFailed]fail to split region; split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader: : [BR:Restore:ErrRestoreSplitFailed]fail to split region"] [errorVerbose="the following errors occurred:
- [BR:Restore:ErrRestoreSplitFailed]fail to split region
split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader:
github.com/pingcap/tidb/br/pkg/restore/split.sendSplitRegionRequest
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/client.go:370
github.com/pingcap/tidb/br/pkg/restore/split.(*pdClient).sendSplitRegionRequest
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/client.go:315
github.com/pingcap/tidb/br/pkg/restore/split.(*pdClient).BatchSplitRegionsWithOrigin
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/client.go:413
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).BatchSplitRegions
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/localhelper.go:400
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).SplitAndScatterRegionByRanges.func3
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/localhelper.go:276
golang.org/x/sync/errgroup.(*Group).Go.func1
/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.