DDL: DROP DATABASE on a large schema can OOM TiDB and leave the DDL job stuck in queueing
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
This is based on a sanitized real-world observation. The exact minimal synthetic reproducer still needs to be built.
Observed pattern:
1. Create or use a very large schema with tens of thousands of tables. In the observed case, a read against `information_schema.tables` for the target schema returned `47,991` rows, indicating about `47,991` tables in that schema.
2. Run:
```sql
DROP DATABASE IF EXISTS ;
```
3. Watch TiDB DDL progress and process memory.
Relevant observations from the incident:
- The target schema had about `47,991` tables according to `information_schema.tables`.
- The `DROP DATABASE` DDL job was submitted successfully and stayed in `State:queueing` with `Err:` and `ErrCount:0`.
- DDL workers repeatedly logged `run one job step` for the same `DROP DATABASE` job, but the persisted job state still printed as `queueing`.
- TiDB memory quickly grew close to the process/container memory limit. Example observed values were roughly 4.3-4.9 GiB RSS with a visible quota around 5 GiB.
- TiDB repeatedly restarted, causing DDL owner churn. Each new owner retried the job, but the job remained `queueing`.
- The `DROP DATABASE` statement itself did not show tracked SQL memory usage in the expensive-query log (`mem_max` was reported as 0), and no single slow-query memory consumer exceeded TiDB's per-session global-memory kill threshold.
- TiDB's global memory controller repeatedly logged that it tried to kill the top memory consumer but found no session larger than `tidb_server_memory_limit_sess_min_size`.
This suggests the memory pressure may come from untracked or aggregate internal memory, likely metadata/InfoSchema/catalog handling for a very large schema rather than a single SQL session.
Code paths that look relevant in v26.3.4:
- `pkg/ddl/job_worker.go`: `runOneJobStep` logs the job before the in-memory state is changed from `queueing` to `running` and before the updated job is persisted.
- `pkg/ddl/job_worker.go`: the persisted transition happens later through `updateDDLJob` and transaction commit. If the owner restarts or loses ownership before commit, the job can remain visibly `queueing`.
- `pkg/ddl/schema.go`: `onDropSchema` calls `metaMut.ListTables` while handling `DROP DATABASE`.
- `pkg/meta/meta.go`: `ListTables` materializes table metadata for the schema.
Hypothesis: for very large schemas, `DROP DATABASE` or repeated owner startup/schema reload work may allocate enough untracked/aggregate memory to restart TiDB before the first DDL job state update is committed. The job then remains stuck in `queueing` and retries under new owners.
### 2. What did you expect to see? (Required)
`DROP DATABASE` on a large schema should either:
- make forward progress without restarting TiDB, or
- fail/cancel with an actionable DDL error, or
- apply memory controls/backpressure so TiDB remains alive and the DDL owner can persist progress.
The DDL job should not remain indefinitely in `queueing` with no error while TiDB repeatedly restarts.
### 3. What did you see instead (Required)
The `DROP DATABASE` job stayed in `State:queueing` for hours. TiDB repeatedly hit memory pressure and restarted. DDL ownership moved repeatedly, and every new owner retried the same job while it still printed as `queueing`.
No earlier DDL job was found to explain the queueing state, and the DDL job did not report a job-level error. The issue looked like a restart-before-commit loop: the worker starts a DDL step, memory pressure/restart happens before the queueing-to-running state transition is persisted, and the next owner observes the same queued job again.
### 4. What is your TiDB version? (Required)
Observed on TiDB `v26.3.4`:
```text
Release Version: v26.3.4
Git Commit Hash: d6e91f7689a753ef0a82453418f76fe843ce0293
Build Time: 2026-06-18 10:28:38 UTC
```
Contributor guide
Assessment
This issue has not been assessed yet.