ClickHouse / ClickHouse/ClickHouse
Crash between the mutation entry and the metadata commit of `ALTER ... MODIFY TTL` wedges the table's mutation queue permanently
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
ClickHouse (found by an internal crash-durability test framework)
### Describe what's wrong
On plain (non-replicated) `MergeTree`, `ALTER TABLE ... MODIFY TTL` commits two independent on-disk records with no atomicity between them: the mutation entry `mutation_.txt` holding `(MATERIALIZE TTL)`, and the table metadata `.sql` that *defines* the TTL the mutation is supposed to materialize.
A crash after the first and before the second leaves a mutation that can never succeed. On restart the half-written metadata is discarded, so the table has no TTL, while the surviving `mutation_.txt` still says `MATERIALIZE TTL`. The background executor then retries it forever:
```
Code: 80. DB::Exception: Cannot MATERIALIZE TTL as there is no TTL set for table default.ttlw_t. (INCORRECT_QUERY)
```
The damage is not to this mutation but to **every later mutation on the table**. Plain `MergeTree` applies a part's mutations as a set up to a version, so a subsequent unrelated statement — here `ALTER TABLE ttlw_t UPDATE val = val + 1 WHERE id < 10` — is attempted as part `all_1_6_1_8`, hits the same impossible `MATERIALIZE TTL`, and never completes either. It also reports the TTL error, which has nothing to do with the statement the user issued:
```sql
SELECT mutation_id, command, is_done, substring(latest_fail_reason, 1, 60) FROM system.mutations WHERE table = 'ttlw_t';
┌─mutation_id─────┬─command──────────────────────────────┬─is_done─┬─substring(latest_fail_reason, 1, 60)─────────────────────────┐
│ mutation_7.txt │ (MATERIALIZE TTL) │ 0 │ Code: 80. DB::Exception: Cannot MATERIALIZE TTL as there is │
│ mutation_8.txt │ (UPDATE val = val + 1 WHERE id < 10) │ 0 │ Code: 80. DB::Exception: Cannot MATERIALIZE TTL as there is │
└─────────────────┴──────────────────────────────────────┴─────────┴──────────────────────────────────────────────────────────────┘
```
The table's mutation queue is therefore wedged from the moment of the crash, and **restarting the server does not clear it** — the entry is on disk, so it is reloaded and fails again. No data is lost or corrupted; the failure mode is liveness, and it is silent apart from repeated `` lines in the server log.
`KILL MUTATION WHERE table = 'ttlw_t' AND is_done = 0` does clear it, so an operator who notices can recover. Two things make noticing hard: nothing surfaces above the log unless `system.mutations` is inspected, and the error text points at TTL while the statement the user is waiting on is an ordinary `UPDATE`.
`ReplicatedMergeTree` and `SharedMergeTree` are not affected — their mutations and metadata are Keeper transactions.
**Relationship to #113459.** That issue reports the same commit window — mutation entry durable before the metadata commit — for `ALTER ... RENAME COLUMN`, where the orphan mutation *applies* and silently returns defaults. This is the opposite outcome of the same window: a mutation that can never apply and blocks the queue behind it. I am filing separately because the symptom, the severity axis and the search terms are entirely different, and because a fix scoped to `RENAME COLUMN` would leave this live. If you prefer one issue per root cause, please close this as a duplicate — a single atomicity fix for the pair would cover both.
### Does it reproduce on the most recent release?
Yes, reproduced on `26.8.1.1` (current master build).
### How to reproduce
Reproduced with a SIGKILL delivered between the two commit records; no power loss, no special filesystem and **no non-default settings** are required — verified at production defaults as well as with the fsync family pinned on, with identical results (2 of 6 trials each).
1. Start a plain single-node server.
2. Create a table with **no** TTL and insert some rows:
```sql
CREATE TABLE ttlw_t (id UInt64, val UInt64, ts DateTime) ENGINE = MergeTree ORDER BY id;
INSERT INTO ttlw_t SELECT number, number * 13, now() FROM numbers(300);
```
3. Issue the TTL alter, concurrently with `OPTIMIZE TABLE ttlw_t FINAL` in a loop (the concurrent rename traffic is only there to make the crash land inside the DDL commit window reliably):
```sql
ALTER TABLE ttlw_t MODIFY TTL ts + INTERVAL 3 YEAR;
```
4. `SIGKILL` the server during that window — specifically after `mutation_.txt` has been written and before `.sql.tmp` is renamed onto `.sql`.
5. Restart. The schema has no TTL, the mutation is still there, and it fails on every retry:
```sql
SHOW CREATE TABLE ttlw_t; -- no TTL clause
SELECT command, is_done, latest_fail_reason FROM system.mutations WHERE table = 'ttlw_t';
```
6. Issue any ordinary mutation and observe that it never finishes:
```sql
ALTER TABLE ttlw_t UPDATE val = val + 1 WHERE id < 10;
```
Recovered on-disk state and log lines from one reproduction
The interrupted metadata commit is stated plainly at startup:
```
DatabaseAtomic (default): Removing file store/a10//ttlw_t.sql.tmp
```
The surviving `.sql` is the pre-`ALTER` one, with no TTL:
```
ATTACH TABLE _ UUID '16a02dc2-...'
( `id` UInt64, `val` UInt64, `ts` DateTime )
ENGINE = MergeTree
ORDER BY id
SETTINGS index_granularity = 8192
```
while the table's data directory still holds the orphan entry, and then the blocked follow-up:
```
$ cat store/16a/16a02dc2-.../mutation_7.txt
format version: 1
create time: 2026-08-06 04:54:20
commands: (MATERIALIZE TTL)
$ cat store/16a/16a02dc2-.../mutation_8.txt
format version: 1
create time: 2026-08-06 04:54:20
commands: (UPDATE val = val + 1 WHERE id < 10)
```
The executor retries indefinitely — the same task reappears roughly once a second, here on the part that carries both mutation versions:
```
04:54:20.889 {…::all_1_6_1_7} MutatePlainMergeTreeTask: Code: 80 … Cannot MATERIALIZE TTL as there is no TTL set for table default.ttlw_t
04:54:20.963 default.ttlw_t: Added mutation: mutation_8.txt
04:54:20.963 {…::all_1_6_1_8} MutatePlainMergeTreeTask: Code: 80 … Cannot MATERIALIZE TTL as there is no TTL set for table default.ttlw_t
04:54:21.886 {…::all_1_6_1_8} MutatePlainMergeTreeTask: Code: 80 … (same)
04:54:22.729 {…::all_1_6_1_8} MutatePlainMergeTreeTask: Code: 80 … (same)
```
Data is unaffected throughout: 300 rows and `sum(val) = 583050` before the crash and after recovery. After `KILL MUTATION ... SYNC` both entries disappear and the pending `UPDATE` settles (`sum(val) = 583060`).
### Expected behavior
`ALTER TABLE ... MODIFY TTL` should be all-or-nothing across a crash: either the TTL is in the metadata and the `MATERIALIZE TTL` mutation is queued, or neither is present. A mutation entry should never survive a rollback of the metadata that gives it meaning.
Failing that, a mutation whose command is impossible against the current schema should be discarded — with a log message naming it — rather than retried forever, and it should not prevent unrelated later mutations on the same table from completing.
### Error message and/or stacktrace
```
2026.08.06 04:54:20.889390 [ 3825265 ] {16a02dc2-9d38-429e-8f4d-1baf64737aaf::all_1_6_1_7} MutatePlainMergeTreeTask: Code: 80. DB::Exception: Cannot MATERIALIZE TTL as there is no TTL set for table default.ttlw_t (16a02dc2-9d38-429e-8f4d-1baf64737aaf). (INCORRECT_QUERY) (version 26.8.1.1 (official build))
2026.08.06 04:54:20.889703 [ 3825265 ] {} MergeTreeBackgroundExecutor: Exception while executing background task {16a02dc2-9d38-429e-8f4d-1baf64737aaf::all_1_6_1_7}: Code: 80. DB::Exception: Cannot MATERIALIZE TTL as there is no TTL set for table default.ttlw_t (16a02dc2-9d38-429e-8f4d-1baf64737aaf). (INCORRECT_QUERY)
```
### Additional context
Found by a crash-durability test framework and then reduced to a standalone deterministic reproducer, run outside the framework's own judging path (plain SIGKILL through an `LD_PRELOAD` shim, no fault simulator, no oracles) so that a reproduction cannot be an artifact of the harness. 2 of 6 trials reproduce, at production defaults and with the fsync family pinned alike; the other 4 land outside the window and the orphan mutation completes harmlessly.
Related, same family, both open: #113459 (the same commit window with `RENAME COLUMN`, opposite outcome) and #111380 (the mutation entry file's own lifecycle is not crash-durable — notably, an acknowledged `KILL MUTATION`, the recovery action for this issue, can itself silently revert after power loss).
Contributor guide
Assessment
This issue has not been assessed yet.