pingcap / pingcap/tidb

Cancel ADD INDEX GLOBAL (ingest mode) may crash TiDB due to tmp_ddl cleanup removing SST files

Open
#65,958 1 comment 0 reactions 1 assignee Claimed by @fzzf678 View on GitHub
affects-8.5 component/ddl report/customer severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

Observed in production; local reproduction not yet achieved. Attempted steps:

1. Configure `temp-dir=/tmp/tidb` and enable ingest/fast reorg + dist task:
```sql
set global tidb_ddl_enable_fast_reorg = ON;
set global tidb_enable_dist_task = ON;
```
2. Create a partitioned table.
3. Session1: start adding a GLOBAL index on a column of that partitioned table.
```sql
ALTER TABLE ADD INDEX () GLOBAL;
```
4. Session2: cancel the job while it is running:
```sql
ADMIN SHOW DDL JOBS;
ADMIN CANCEL DDL JOBS ;
```
5. Wait ~1 minute (cleanup loop interval).

### 2. What did you expect to see? (Required)

Canceling the DDL job should not crash TiDB; temporary files should be cleaned up safely.

### 3. What did you see instead (Required)

TiDB exited and was restarted by systemd roughly ~1 minute after canceling the DDL. `tidb_stderr.log` shows Pebble fatal due to missing SST:

```
2026/01/23 10:45:41 000005.sst:
orig err: open /tmp/tidb/tmp_ddl-4001/881158/211d99e7-03e4-5ffc-9cd3-476e9437af7c/000005.sst: no such file or directory
list err: open /tmp/tidb/tmp_ddl-4001/881158/211d99e7-03e4-5ffc-9cd3-476e9437af7c: no such file or directory
```

Local reproduction attempts (following the steps in section 1) did not reproduce the crash.

### 4. What is your TiDB version? (Required)

v8.5.4

---

### Additional info / suspected root cause

- Suspected race: the DDL ingest cleanup loop removes `${temp-dir}/tmp_ddl-${port}/` directories based on `mysql.tidb_ddl_job` every 1 minute. After `ADMIN CANCEL DDL`, the job record disappears, but the Lightning local backend may still be closing Pebble engines. The cancel error path only calls `FinishAndUnregisterEngines(OptCloseEngines)` (does not clean data). Cleanup then `RemoveAll` the directory while Pebble background goroutines are still accessing SSTs, leading to Pebble fatal and TiDB exit.
- The fatal message format matches Pebble `MustExist` when a file is missing.

### Why not reproduced (hypotheses)

- Narrow race window: cleanup tick is 1 minute; if engines finish closing before cleanup, no crash.
- Ingest mode might not have been selected (fast reorg disabled or lightning env unavailable), so no local Pebble engines were used.
- Table/index too small, or cancel timing too early/late, so no background Pebble activity when cleanup runs.

Suggested labels: component/ddl, severity/major, may-affects-8.5

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.