ClickHouse / ClickHouse/ClickHouse

Unkillable synchronous DROP of a table with inner tables: the inner drop waits in `waitTableFinallyDropped` without a process-list element (stress hung check: `DROP TABLE prometheus`)

Open
#114,259 4 comments 0 reactions 0 assignees View on GitHub
comp-ddl
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

## Summary

`Stress test (*)` / `Hung check failed, possible deadlock found` has a recurring shape: `DROP TABLE prometheus` (a `TimeSeries` table, from the stateless test `04811_promql_topk_bottomk_limitk.sql`) hangs for more than 1000 s and cannot be killed, and `DROP DATABASE ... SYNC` / `DETACH DATABASE` queries pile up behind it on the `DDLGuard`.

There are two layers:

1. **The synchronous inner-table drop is unkillable.** `StorageTimeSeries::dropInnerTableIfAny` (and the same code path for `StorageMaterializedView` / `StorageWindowView` inner tables) runs the inner `DROP` via `InterpreterDropQuery::executeDropQuery`, which executes it on a copy of the **global** context. That context has no process-list element, so in `InterpreterDropQuery::waitForTableToBeActuallyDroppedOrDetached` the `query_status` is `nullptr`, the `throw_if_cancelled` callback is a no-op, and `DatabaseCatalog::waitTableFinallyDropped` waits on the condition variable forever. `KILL QUERY` on the outer `DROP TABLE` is accepted (`is_cancelled: 1`) but has no effect. The outer query keeps holding the `DDLGuard`, so every later DDL on the same objects queues up behind it — the hung check then reports the whole convoy.

2. **A leaked `StoragePtr` reference to the inner table prevents the final drop.** In the analyzed run the background drop task was alive and dropped the sibling inner tables (`metrics`, `tags`) within 200 ms, but the `samples` inner table stayed "in use" until server shutdown: `DatabaseCatalog: Have 3 tables in drop queue (2 of them are in use), will try drop 1 tables`, with all other threads idle. The extra reference was parked in some long-lived structure by an earlier query of the same stress session (the server-side AST fuzzer had replayed `CREATE TABLE prometheus__fuzz_N UUID '...' ... SAMPLES INNER UUID '...'` clones with the *same* UUIDs and mangled `prometheusQuery` selects). A plain replay of the whole test with `database_atomic_wait_for_drop_and_detach_synchronously = 1` drops cleanly, so the leak needs the fuzzed queries; the exact holder is not identified yet.

Layer 1 turns any such stuck reference from a transient error into a permanent unkillable hang and a red `Hung check`. With layer 1 fixed, `KILL QUERY` (or `max_execution_time`) interrupts the wait with `QUERY_WAS_CANCELLED`, the `DDLGuard` is released, and the convoy dissolves.

## Evidence

Timeline from `Stress test (arm_release)` on #114073 (commit `848c4b6b`), query id `f4c3682d-7b10-4c94-948b-615a8b5277cb`:

```text
03:42:32.503 {f4c3682d...} DROP TABLE prometheus; (stage: Complete)
03:42:32.523 {f4c3682d...} DatabaseAtomic (test_q5t0a4z4u2l8): Counting detached table .inner_id.samples.a27aed32-...
03:42:32.530 {f4c3682d...} DatabaseCatalog: Waiting for table 76f6c994-1aaf-4fa0-bdc5-aaf487998b12 to be finally dropped
03:42:44.490 KillQuery: Will kill query f4c3682d-7b10-4c94-948b-615a8b5277cb (asynchronously)
...
03:52..shutdown, repeating: DatabaseCatalog: Have 3 tables in drop queue (2 of them are in use), will try drop 1 tables
```

The hung thread's stack: `InterpreterDropQuery::executeToTableImpl` → `DatabaseAtomic::dropTable` → `StorageTimeSeries::dropInnerTableIfAny` → inner `InterpreterDropQuery::execute` → `waitForTableToBeActuallyDroppedOrDetached` → `DatabaseCatalog::waitTableFinallyDropped` (condvar, forever). The queued `DROP DATABASE ... SYNC` / `DETACH DATABASE` stacks end in `DatabaseCatalog::getDDLGuard`.

Reports:
- https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=114073&sha=848c4b6b912cb20a31d7931ff22d376f6975f7de&name_0=PR&name_1=Stress%20test%20%28arm_release%29
- https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=114073&sha=848c4b6b912cb20a31d7931ff22d376f6975f7de&name_0=PR&name_1=Stress%20test%20%28arm_debug%29
- The same failure on master (so it is not related to that PR): https://s3.amazonaws.com/clickhouse-test-reports/json.html?REF=master&sha=7fc48be1accbc1d4cdb994c0db6af5d977d3910f&name_0=MasterCI&name_1=Stress%20test%20%28arm_asan_ubsan%2C%20s3%29
- The same hung check (with `prometheus` in the context) hit at least 6 unrelated PRs on 2026-08-10 alone: #111973, #107669, #112469, #107179, #112088, #114054.

The test `04811_promql_topk_bottomk_limitk.sql` landed on master on 2026-08-08, which matches the first occurrences of this shape.

## Fix directions

1. Propagate the original query's process-list element into the inner-drop context in `InterpreterDropQuery::executeDropQuery`, so the synchronous inner-drop wait becomes killable (same treatment as `waitTableFinallyDropped`'s existing `throw_if_cancelled` hook and the `UNDROP TABLE` fix in #113263).
2. Find the `StoragePtr` leak of the `TimeSeries` inner table under the fuzzer's same-UUID `CREATE`/`prometheusQuery` replay. Adding diagnostics to `DatabaseCatalog::dropTableDataTask` (log the table name and time-in-queue for entries that stay "in use" longer than the drop delay) would identify the holder on the next CI occurrence.

Related: https://github.com/ClickHouse/ClickHouse/pull/113263
Related: https://github.com/ClickHouse/ClickHouse/pull/114073

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.