pingcap / pingcap/tidb

Slow IMPORT DATA Performance: 3 Hours for 45GB Dataset

Open
#63,461 1 comment 0 reactions 0 assignees View on GitHub
contribution
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

We are experiencing significantly slower than expected import speeds when using the IMPORT INTO command to load specific CSV datasets into TiDB. The performance degradation is noticeable compared to other datasets of similar size and structure.

### 1. Minimal reproduce step (Required)

Execute an import on an internal dataset (approximately 45GB of data size, 77GB of index size), and then run SHOW IMPORT JOBS.

```
MySQL [(none)]> show import jobs;
+--------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------+----------+----------------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+---------------+----------------------------+----------+-------------------------+---------------------+-----------------------+----------------+--------------+
| Job_ID | Group_Key | Data_Source | Target_Table | Table_ID | Phase | Status | Source_File_Size | Imported_Rows | Result_Message | Create_Time | Start_Time | End_Time | Created_By | Last_Update_Time | Cur_Step | Cur_Step_Processed_Size | Cur_Step_Total_Size | Cur_Step_Progress_Pct | Cur_Step_Speed | Cur_Step_ETA |
+--------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------+----------+----------------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+---------------+----------------------------+----------+-------------------------+---------------------+-----------------------+----------------+--------------+
...
| 18 | NULL | ****** | `**`.`*` | 46 | global-sorting | running | 12.57GiB | 0 | | 2025-09-11 02:49:27.387068 | 2025-09-11 02:54:38.581194 | NULL | **** | 2025-09-11 03:53:42 | encode | 11.82GiB | 12.57GiB | 94 | 3.604MiB/s | 00:03:32 |
| 19 | NULL | ****** | `**`.`**` | 48 | global-sorting | running | 15.92GiB | 0 | | 2025-09-11 02:49:29.620143 | 2025-09-11 02:55:36.552449 | NULL | **** | 2025-09-11 03:53:43 | encode | 15.59GiB | 15.92GiB | 97 | 4.907MiB/s | 00:01:08 |
| 20 | NULL | ****** | `**`.`***` | 50 | global-sorting | running | 29.3GiB | 0 | | 2025-09-11 02:49:31.376228 | 2025-09-11 02:55:48.565148 | NULL | **** | 2025-09-11 03:53:44 | encode | 11.3GiB | 29.3GiB | 38 | 2.312MiB/s | 02:12:48 |
```

### 2. What did you expect to see? (Required)

The import operation is expected to complete within a few minutes.

### 3. What did you see instead (Required)

The import operation actually takes approximately 3 hours to complete.

### 4. What is your TiDB version? (Required)

------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v9.0.0-beta.2.pre-479-g5cb0037
Edition: Enterprise
Git Commit Hash: 5cb0037c0fc9b04d594bfd3df28bbe3eb1f8ccb9
Git Branch: HEAD
UTC Build Time: 2025-09-10 20:59:56
GoVersion: go1.23.12
Race Enabled: false
Check Table Before Drop: false
Store: tikv
Enterprise Extension Commit Hash: dc633aae52eb11b4e3549e3a7ef5b0ed14e159b1
Kernel Type: Next Generation |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.002 sec)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the IMPORT INTO operation with the reported dataset and monitor it using SHOW IMPORT JOBS, comparing the slow files or tables with the expected behavior. Done means identifying the cause of the roughly three-hour runtime and providing a fix or documented resolution that brings comparable imports closer to the expected completion time.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.