pingcap / pingcap/tidb

DDL: DROP DATABASE on a large schema can OOM TiDB and leave the DDL job stuck in queueing

Open
#69,534 0 comments 0 reactions 1 assignee Claimed by @D3Hunter View on GitHub
component/ddl component/infoschema may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

This is based on a sanitized real-world observation. The exact minimal synthetic reproducer still needs to be built.

Observed pattern:

1. Create or use a very large schema with tens of thousands of tables. In the observed case, a read against `information_schema.tables` for the target schema returned `47,991` rows, indicating about `47,991` tables in that schema.
2. Run:

```sql
DROP DATABASE IF EXISTS ;
```

3. Watch TiDB DDL progress and process memory.

Relevant observations from the incident:

- The target schema had about `47,991` tables according to `information_schema.tables`.
- The `DROP DATABASE` DDL job was submitted successfully and stayed in `State:queueing` with `Err:` and `ErrCount:0`.
- DDL workers repeatedly logged `run one job step` for the same `DROP DATABASE` job, but the persisted job state still printed as `queueing`.
- TiDB memory quickly grew close to the process/container memory limit. Example observed values were roughly 4.3-4.9 GiB RSS with a visible quota around 5 GiB.
- TiDB repeatedly restarted, causing DDL owner churn. Each new owner retried the job, but the job remained `queueing`.
- The `DROP DATABASE` statement itself did not show tracked SQL memory usage in the expensive-query log (`mem_max` was reported as 0), and no single slow-query memory consumer exceeded TiDB's per-session global-memory kill threshold.
- TiDB's global memory controller repeatedly logged that it tried to kill the top memory consumer but found no session larger than `tidb_server_memory_limit_sess_min_size`.

This suggests the memory pressure may come from untracked or aggregate internal memory, likely metadata/InfoSchema/catalog handling for a very large schema rather than a single SQL session.

Code paths that look relevant in v26.3.4:

- `pkg/ddl/job_worker.go`: `runOneJobStep` logs the job before the in-memory state is changed from `queueing` to `running` and before the updated job is persisted.
- `pkg/ddl/job_worker.go`: the persisted transition happens later through `updateDDLJob` and transaction commit. If the owner restarts or loses ownership before commit, the job can remain visibly `queueing`.
- `pkg/ddl/schema.go`: `onDropSchema` calls `metaMut.ListTables` while handling `DROP DATABASE`.
- `pkg/meta/meta.go`: `ListTables` materializes table metadata for the schema.

Hypothesis: for very large schemas, `DROP DATABASE` or repeated owner startup/schema reload work may allocate enough untracked/aggregate memory to restart TiDB before the first DDL job state update is committed. The job then remains stuck in `queueing` and retries under new owners.

### 2. What did you expect to see? (Required)

`DROP DATABASE` on a large schema should either:

- make forward progress without restarting TiDB, or
- fail/cancel with an actionable DDL error, or
- apply memory controls/backpressure so TiDB remains alive and the DDL owner can persist progress.

The DDL job should not remain indefinitely in `queueing` with no error while TiDB repeatedly restarts.

### 3. What did you see instead (Required)

The `DROP DATABASE` job stayed in `State:queueing` for hours. TiDB repeatedly hit memory pressure and restarted. DDL ownership moved repeatedly, and every new owner retried the same job while it still printed as `queueing`.

No earlier DDL job was found to explain the queueing state, and the DDL job did not report a job-level error. The issue looked like a restart-before-commit loop: the worker starts a DDL step, memory pressure/restart happens before the queueing-to-running state transition is persisted, and the next owner observes the same queued job again.

### 4. What is your TiDB version? (Required)

Observed on TiDB `v26.3.4`:

```text
Release Version: v26.3.4
Git Commit Hash: d6e91f7689a753ef0a82453418f76fe843ce0293
Build Time: 2026-06-18 10:28:38 UTC
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.