pingcap / pingcap/tidb

Region is not balanced when Lighting import million tables

Open
#58,508 2 comments 0 reactions 0 assignees View on GitHub
component/lightning may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

6 tikv
Lighting import 1 million tables (1w rows each table)
![Image](https://github.com/user-attachments/assets/346c0546-c69d-43af-b23e-8e73db5f44f2)

```toml
### tidb-lightning configuration
[lightning]
# check if the cluster satisfies the minimum requirement before starting
check-requirements = false
index-concurrency = 16
table-concurrency = 16

[mydumper]
no-schema = true
case-sensitive = false
data-source-dir = "xxx"

# https://docs.pingcap.com/zh/tidb/stable/tidb-lightning-configuration
[conflict]
strategy = ""

[tikv-importer]
backend = "local"
sorted-kv-dir = "/data/sorted-kv-dir"
# parallel-import = true
disk-quota = "3TB"
range-concurrency = 4

# CSV files are imported according to MySQL LOAD DATA INFILE rules.
[mydumper.csv]
# separator between fields, should be an ASCII character.
separator = ","
# whether the CSV files contain a header. If true, the first line will be skipped
header = true
# whether the CSV contains any NULL value. If true, all columns from CSV cannot be NULL.
not-null = false
# if non-null = false (i.e. CSV can contain NULL), fields equal to this value will be treated as NULL
null = "NULL"
# whether to interpret backslash-escape inside strings.
backslash-escape = false
# if a line ends with a separator, remove it.
trim-last-separator = false

[checkpoint]
enable = true

# post-restore provide some options which will be executed after all kv data has been imported into the tikv cluster.
# the execution order are(if set true): checksum -> analyze
[post-restore]
# if set true, checksum will do ADMIN CHECKSUM TABLE for each table.
# checksum = false
# if set true, analyze will do ANALYZE TABLE for each table.
analyze = false

[tidb]
checksum-table-concurrency = 16
```
### 2. What did you expect to see? (Required)

### 3. What did you see instead (Required)

### 4. What is your TiDB version? (Required)

Release Version: v8.5.0
Git Commit Hash: cf1bb564d4c029a27049567fae4457c4d985720b
Git Branch: heads/refs/tags/v8.5.0
Go Version: go1.23.4
UTC Build Time: 2024-12-12 16:11:49
Race Enabled: false

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.