orphaned backfill tasks after parent DDL commands cancelled
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
The issue I'm experiencing is that a DDL command is stuck in a pending state because it appears some older DDL commands (nearly a month old) never completed
I do not know what the commands were, but I probably cancelled them weeks ago because they were themselves taking too long to run.
### 2. What did you expect to see? (Required)
I expected my query (an `alter` query to add an index to a table) to run quickly and complete within seconds.
### 3. What did you see instead (Required)
Instead, it was stuck in a pending backfill state, and when I checked, I found 8 older backfill commands still in the queue:
```
MySQL [user5124]> SELECT id, task_key, type, state, create_time, start_time, end_time FROM mysql.tidb_global_task ORDER BY create_time DESC LIMIT 50;
+--------+----------------------+----------+---------+---------------------+---------------------+----------+
| id | task_key | type | state | create_time | start_time | end_time |
+--------+----------------------+----------+---------+---------------------+---------------------+----------+
| 120004 | ddl/backfill/1081941 | backfill | pending | 2025-11-14 09:39:06 | NULL | NULL |
| 120003 | ddl/backfill/1076035 | backfill | pending | 2025-10-25 16:28:37 | NULL | NULL |
| 120001 | ddl/backfill/1076027 | backfill | pending | 2025-10-25 12:25:33 | NULL | NULL |
| 90021 | ddl/backfill/1069437 | backfill | running | 2025-10-06 07:26:58 | 2025-10-08 13:26:55 | NULL |
| 90012 | ddl/backfill/1069428 | backfill | running | 2025-10-06 07:21:42 | 2025-10-08 13:26:53 | NULL |
| 90011 | ddl/backfill/1069426 | backfill | running | 2025-10-06 07:21:15 | 2025-10-08 10:22:25 | NULL |
| 60115 | ddl/backfill/1069409 | backfill | running | 2025-10-04 11:41:37 | 2025-10-06 08:21:56 | NULL |
| 60068 | ddl/backfill/1067586 | backfill | running | 2025-10-03 15:43:39 | 2025-10-03 15:43:40 | NULL |
| 30001 | ddl/backfill/1067184 | backfill | running | 2025-10-03 09:30:09 | 2025-10-03 09:30:10 | NULL |
+--------+----------------------+----------+---------+---------------------+---------------------+----------+
```
cancelling the older DDL commands is not working because they do not appear to exist anymore:
```
MySQL [user5124]> admin show ddl jobs where job_id in (1076035,1076027,1069437,1069428,1069426,1069409,1067586,1067184);
Empty set (0.040 sec)
```
To complete my re-indexing, I had to cancel the stuck DDL command, disable re-org and DXF and re-run the ALTER command:
```
MySQL [user5124]> SET GLOBAL tidb_ddl_enable_fast_reorg = OFF;
Query OK, 0 rows affected (0.041 sec)
MySQL [user5124]> SET GLOBAL tidb_enable_dist_task = OFF;
Query OK, 0 rows affected (0.035 sec)
MySQL [user5124]> alter table user5124_tasks add key key4 (status,assigned_to);
Query OK, 0 rows affected (2.933 sec)
```
### 4. What is your TiDB version? (Required)
```
MySQL [user5124]> select tidb_version();
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version() |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v8.5.3
Edition: Community
Git Commit Hash: dc2548aac79a712265e831cff2a3a896bc0a5a38
Git Branch: HEAD
UTC Build Time: 2025-07-31 13:54:43
GoVersion: go1.23.8
Race Enabled: false
Check Table Before Drop: false
Store: tikv |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.025 sec)
```
Contributor guide
Assessment
This issue has not been assessed yet.