pingcap / pingcap/tidb

v6.1.4, lightning too slow when import 20Gb data

Open
#40,960 2 comments 0 reactions 0 assignees View on GitHub
component/lightning found/automation
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

1. Create tidb cluster
2. Scale-out tidb,tikv,pd
3. Use lightning to import
4. Scale-in tidb,tikv,pd
5. Use lightning to import again
6. Check the data consistency
--------above passed
8. use lightning import 20Gb to upstream cluster, lightning hang
{"command": "/tidb-lightning \"-pd-urls\" \"src-tidb-pd.jianxu-testbed-mxts7:2379\" \"-tidb-host\" \"src-tidb-tidb.jianxu-testbed-mxts7\" \"-tidb-port\" \"4000\" \"-tidb-user\" \"root\" \"-tidb-password\" \"\" \"-backend\" \"local\" \"-sorted-kv-dir\" \"/tmp/sorted-kv-dir\" \"-d\" \"s3://nfs/tools_test_data/csv_export_20GB?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2fminio.pingcap.net%3a9001&force-path-style=true\" \"-c\" \"/tmp/tidb-lightning.toml\"",

### 2. What did you expect to see? (Required)
lightning import successfully

### 3. What did you see instead (Required)
process is keep 0%

### 4. What is your TiDB version? (Required)

[2023/02/01 13:08:21.564 +00:00] [INFO] [info.go:49] ["Welcome to TiDB-Lightning"] [release-version=v6.1.4] [git-hash=ac6560f9b7239f0ed3f8c39482b79b48a6263de1] [git-branch=

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.