v6.1.4, lightning too slow when import 20Gb data
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
1. Create tidb cluster
2. Scale-out tidb,tikv,pd
3. Use lightning to import
4. Scale-in tidb,tikv,pd
5. Use lightning to import again
6. Check the data consistency
--------above passed
8. use lightning import 20Gb to upstream cluster, lightning hang
{"command": "/tidb-lightning \"-pd-urls\" \"src-tidb-pd.jianxu-testbed-mxts7:2379\" \"-tidb-host\" \"src-tidb-tidb.jianxu-testbed-mxts7\" \"-tidb-port\" \"4000\" \"-tidb-user\" \"root\" \"-tidb-password\" \"\" \"-backend\" \"local\" \"-sorted-kv-dir\" \"/tmp/sorted-kv-dir\" \"-d\" \"s3://nfs/tools_test_data/csv_export_20GB?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2fminio.pingcap.net%3a9001&force-path-style=true\" \"-c\" \"/tmp/tidb-lightning.toml\"",
### 2. What did you expect to see? (Required)
lightning import successfully
### 3. What did you see instead (Required)
process is keep 0%
### 4. What is your TiDB version? (Required)
[2023/02/01 13:08:21.564 +00:00] [INFO] [info.go:49] ["Welcome to TiDB-Lightning"] [release-version=v6.1.4] [git-hash=ac6560f9b7239f0ed3f8c39482b79b48a6263de1] [git-branch=
Contributor guide
Assessment
This issue has not been assessed yet.