Cancel ADD INDEX GLOBAL (ingest mode) may crash TiDB due to tmp_ddl cleanup removing SST files
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
Observed in production; local reproduction not yet achieved. Attempted steps:
1. Configure `temp-dir=/tmp/tidb` and enable ingest/fast reorg + dist task:
```sql
set global tidb_ddl_enable_fast_reorg = ON;
set global tidb_enable_dist_task = ON;
```
2. Create a partitioned table.
3. Session1: start adding a GLOBAL index on a column of that partitioned table.
```sql
ALTER TABLE ADD INDEX () GLOBAL;
```
4. Session2: cancel the job while it is running:
```sql
ADMIN SHOW DDL JOBS;
ADMIN CANCEL DDL JOBS ;
```
5. Wait ~1 minute (cleanup loop interval).
### 2. What did you expect to see? (Required)
Canceling the DDL job should not crash TiDB; temporary files should be cleaned up safely.
### 3. What did you see instead (Required)
TiDB exited and was restarted by systemd roughly ~1 minute after canceling the DDL. `tidb_stderr.log` shows Pebble fatal due to missing SST:
```
2026/01/23 10:45:41 000005.sst:
orig err: open /tmp/tidb/tmp_ddl-4001/881158/211d99e7-03e4-5ffc-9cd3-476e9437af7c/000005.sst: no such file or directory
list err: open /tmp/tidb/tmp_ddl-4001/881158/211d99e7-03e4-5ffc-9cd3-476e9437af7c: no such file or directory
```
Local reproduction attempts (following the steps in section 1) did not reproduce the crash.
### 4. What is your TiDB version? (Required)
v8.5.4
---
### Additional info / suspected root cause
- Suspected race: the DDL ingest cleanup loop removes `${temp-dir}/tmp_ddl-${port}/` directories based on `mysql.tidb_ddl_job` every 1 minute. After `ADMIN CANCEL DDL`, the job record disappears, but the Lightning local backend may still be closing Pebble engines. The cancel error path only calls `FinishAndUnregisterEngines(OptCloseEngines)` (does not clean data). Cleanup then `RemoveAll` the directory while Pebble background goroutines are still accessing SSTs, leading to Pebble fatal and TiDB exit.
- The fatal message format matches Pebble `MustExist` when a file is missing.
### Why not reproduced (hypotheses)
- Narrow race window: cleanup tick is 1 minute; if engines finish closing before cleanup, no crash.
- Ingest mode might not have been selected (fast reorg disabled or lightning env unavailable), so no local Pebble engines were used.
- Table/index too small, or cancel timing too early/late, so no background Pebble activity when cleanup runs.
Suggested labels: component/ddl, severity/major, may-affects-8.5
Contributor guide
Assessment
This issue has not been assessed yet.