pingcap / pingcap/tidb

lightning parallel import cause region count grows

Open
#58,786 1 comment 0 reactions 1 assignee Claimed by @OliverS929 View on GitHub
component/lightning severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

use multiple lightning node to import

### 2. What did you expect to see? (Required)

the region count should be almost the same as there's only one lightning node

### 3. What did you see instead (Required)

the region count is #lightning times than the single lightning import

### 4. What is your TiDB version? (Required)

because lightning will call PD `SetRegionLabelRule`, and this API will split region on the request key range. And lightning uses actual data range as the key range. When there're multiple lightning node, each node has different key range. So each node will causes region split once

https://github.com/pingcap/tidb/blob/6c9ee36e8e7c86abbda374d841b019979ff3aaf4/pkg/lightning/backend/local/local.go#L1331-L1343

maybe consider to use the whole range of tableID, or tableID + record/ indexID in the request

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.