pingcap / pingcap/tidb

lightning: skewed region/traffic during import

Open
#45,924 1 comment 0 reactions 1 assignee Claimed by @Benjamin2037 View on GitHub
component/lightning severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

1. drop tables and hot write/read to make PD busy with scheduling
2. lightning start to import data
3. manually use pd-ctl to cancel some `scatter-region` operators

### 2. What did you expect to see? (Required)
import CPU/trafics/regions in tikv was balanced

### 3. What did you see instead (Required)
import CPU skewed(skewed trafic and regions)

### 4. What is your TiDB version? (Required)
v6.5.0~master

https://github.com/pingcap/tidb/blob/cf441574864be63938524e7dfcf7cc659edc3dd8/br/pkg/lightning/backend/local/localhelper.go#L515-L545

Here, lightning attempts to verify the status of `scatter-region` for the special region in PD. However, it assumes that `scatter-region` is successful when:
- if the region is not found
- if the operator on current region was not scatter-region
Then the above assumption does not hold true in the case of a busy schedule in the current region. One of the simplest ways is to directly use `pd-ctl` to cancel the created scatter-region operator or remove the region directly. In this case, lightning will still consider the `scatter-region` successful.

I think we need to add a new interface for querying the execution results of `scatter-region` in PD instead of guess here

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.