pingcap / pingcap/tidb

lightning Import Large number of tables is too slow

Open
#48,023 2 comments 0 reactions 0 assignees View on GitHub
affects-7.5 affects-8.1 affects-8.5 component/lightning may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
1. lightning import about 3.2w tables (2.7T) to tidb cluster

### 2. What did you expect to see? (Required)
1. finish import finished with Reasonable time
### 3. What did you see instead (Required)
1. It's takes 73h to import the data
2. ddl execute during Import data
```sql
+---------+--------------------+--------------+--------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+--------+
| JOB_ID | DB_NAME | TABLE_NAME | JOB_TYPE | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | CREATE_TIME | START_TIME | END_TIME | STATE |
+---------+--------------------+--------------+--------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+--------+
| 1012315 | test | t4 | create table | public | 2 | 1012314 | 0 | 2023-10-27 02:32:52 | 2023-10-27 02:32:52 | 2023-10-27 02:32:52 | synced |
| 1008704 | test | v0 | create view | public | 2 | 1008703 | 0 | 2023-10-25 14:19:47 | 2023-10-25 14:19:47 | 2023-10-25 14:19:47 | synced |
| 1008702 | test | t1 | create table | public | 2 | 1008701 | 0 | 2023-10-25 14:19:47 | 2023-10-25 14:19:47 | 2023-10-25 14:19:47 | synced |
| 1008700 | test | t0 | create table | public | 2 | 1008699 | 0 | 2023-10-25 14:19:46 | 2023-10-25 14:19:46 | 2023-10-25 14:19:46 | synced |
| 1008698 | test | v0 | drop view | none | 2 | 1008693 | 0 | 2023-10-25 14:19:45 | 2023-10-25 14:19:45 | 2023-10-25 14:19:45 | synced |
| 1008697 | test | t0 | drop table | none | 2 | 1008691 | 0 | 2023-10-25 14:19:42 | 2023-10-25 14:19:42 | 2023-10-25 14:19:43 | synced |
| 1008696 | test | t1 | drop table | none | 2 | 1006140 | 0 | 2023-10-25 14:19:32 | 2023-10-25 14:19:32 | 2023-10-25 14:19:33 | synced |
| 1008694 | test | v0 | create view | public | 2 | 1008693 | 0 | 2023-10-25 14:19:17 | 2023-10-25 14:19:17 | 2023-10-25 14:19:17 | synced |
| 1008692 | test | t0 | create table | public | 2 | 1008691 | 0 | 2023-10-25 14:19:16 | 2023-10-25 14:19:16 | 2023-10-25 14:19:16 | synced |
| 1006145 | test | t3 | create table | public | 2 | 1006144 | 0 | 2023-10-25 09:23:51 | 2023-10-25 09:23:51 | 2023-10-25 09:23:51 | synced |
| 1006143 | test | t2 | create table | public | 2 | 1006142 | 0 | 2023-10-25 09:23:50 | 2023-10-25 09:23:50 | 2023-10-25 09:23:50 | synced |
| 1006141 | test | t1 | create table | public | 2 | 1006140 | 0 | 2023-10-25 09:23:50 | 2023-10-25 09:23:50 | 2023-10-25 09:23:50 | synced |
| 987070 | test | ios3 | create table | public | 2 | 986869 | 0 | 2023-10-23 11:26:50 | 2023-10-23 11:26:50 | 2023-10-23 11:26:51 | synced |
| 986172 | lightning_metadata | task_meta_v2 | create table | public | 254121 | 986171 | 0 | 2023-10-23 09:22:52 | 2023-10-23 09:22:52 | 2023-10-23 09:22:52 | synced |
| 986170 | co01db99 | co01table66 | create table | public | 861939 | 986169 | 0 | 2023-10-23 09:22:41 | 2023-10-23 09:22:42 | 2023-10-23 09:22:42 | synced |
| 986168 | co01db99 | co01table32 | create table | public | 861939 | 986167 | 0 | 2023-10-23 09:22:41 | 2023-10-23 09:22:42 | 2023-10-23 09:22:42 | synced |
| 986166 | co01db99 | co01table69 | create table | public | 861939 | 986165 | 0 | 2023-10-23 09:22:41 | 2023-10-23 09:22:42 | 2023-10-23 09:22:42 | synced |
| 986164 | co01db99 | co01table58 | create table | public | 861939 | 986163 | 0 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | 2023-10-23 09:22:42 | synced |
| 986162 | co01db99 | co01table65 | create table | public | 861939 | 986161 | 0 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | synced |
| 986160 | co01db99 | co01table14 | create table | public | 861939 | 986159 | 0 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | synced |
| 986158 | co01db99 | co01table24 | create table | public | 861939 | 986157 | 0 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | synced |
| 986156 | co01db99 | co01table29 | create table | public | 861939 | 986155 | 0 | 2023-10-23 09:22:40 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | synced |
| 986154 | co01db99 | co01table33 | create table | public | 861939 | 986152 | 0 | 2023-10-23 09:22:40 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | synced |
```
please ignore the `ErrChecksumMismatch` because there are 171 tables is not empty before import.
```log
[2023/10/23 08:54:14.153 +00:00] [INFO] [info.go:49] ["Welcome to TiDB-Lightning"] [release-version=v7.5.0-alpha] [git-hash=79db4ec58be3856995d31013e20f3faa57e3a3ac] [git-branch=heads/refs/tags/v7.5.0-alpha] [go-version=go1.21.3] [utc-build-time="2023-10-16 11:39:02"] [race-enabled=false]
[2023/10/26 11:15:06.489 +00:00] [DEBUG] [import.go:1074] ["update checkpoint"] [table=`co01db99`.`co01table66`] [engine_id=2147483647] [new_status=analyzed] []
[2023/10/26 11:15:07.492 +00:00] [ERROR] [import.go:1650] ["restore all tables data failed"] [takeTime=73h1m3.831306634s] [error="[Lighting:Restore:ErrChecksumMismatch]checksum mismatched remote vs local => (checksum: 11885220305671869244 vs 0) (total_kvs: 300000 vs 600000) (total_bytes:14868675 vs 29737350)"]
[2023/10/26 11:15:07.493 +00:00] [INFO] [import.go:1263] ["everything imported, stopping periodic actions"]
[2023/10/26 11:15:07.494 +00:00] [ERROR] [import.go:558] ["run failed"] [step=4] [error="[Lighting:Restore:ErrChecksumMismatch]checksum mismatched remote vs local => (checksum: 11885220305671869244 vs 0) (total_kvs: 300000 vs 600000) (total_bytes:14868675 vs 29737350)"]
```

### 4. What is your TiDB version? (Required)
master: 79db4ec58be3856995d31013e20f3faa57e3a3ac

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.