ClickHouse / ClickHouse/ClickHouse

Crash between the mutation entry and the metadata commit of `ALTER ... MODIFY TTL` wedges the table's mutation queue permanently

Open
#113,615 1 comment 0 reactions 1 assignee Claimed by @Michicosun View on GitHub
bug comp-mutations comp-ttl
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

ClickHouse (found by an internal crash-durability test framework)

### Describe what's wrong

On plain (non-replicated) `MergeTree`, `ALTER TABLE ... MODIFY TTL` commits two independent on-disk records with no atomicity between them: the mutation entry `mutation_.txt` holding `(MATERIALIZE TTL)`, and the table metadata `.sql` that *defines* the TTL the mutation is supposed to materialize.

A crash after the first and before the second leaves a mutation that can never succeed. On restart the half-written metadata is discarded, so the table has no TTL, while the surviving `mutation_.txt` still says `MATERIALIZE TTL`. The background executor then retries it forever:

```
Code: 80. DB::Exception: Cannot MATERIALIZE TTL as there is no TTL set for table default.ttlw_t. (INCORRECT_QUERY)
```

The damage is not to this mutation but to **every later mutation on the table**. Plain `MergeTree` applies a part's mutations as a set up to a version, so a subsequent unrelated statement — here `ALTER TABLE ttlw_t UPDATE val = val + 1 WHERE id < 10` — is attempted as part `all_1_6_1_8`, hits the same impossible `MATERIALIZE TTL`, and never completes either. It also reports the TTL error, which has nothing to do with the statement the user issued:

```sql
SELECT mutation_id, command, is_done, substring(latest_fail_reason, 1, 60) FROM system.mutations WHERE table = 'ttlw_t';

┌─mutation_id─────┬─command──────────────────────────────┬─is_done─┬─substring(latest_fail_reason, 1, 60)─────────────────────────┐
│ mutation_7.txt │ (MATERIALIZE TTL) │ 0 │ Code: 80. DB::Exception: Cannot MATERIALIZE TTL as there is │
│ mutation_8.txt │ (UPDATE val = val + 1 WHERE id < 10) │ 0 │ Code: 80. DB::Exception: Cannot MATERIALIZE TTL as there is │
└─────────────────┴──────────────────────────────────────┴─────────┴──────────────────────────────────────────────────────────────┘
```

The table's mutation queue is therefore wedged from the moment of the crash, and **restarting the server does not clear it** — the entry is on disk, so it is reloaded and fails again. No data is lost or corrupted; the failure mode is liveness, and it is silent apart from repeated `` lines in the server log.

`KILL MUTATION WHERE table = 'ttlw_t' AND is_done = 0` does clear it, so an operator who notices can recover. Two things make noticing hard: nothing surfaces above the log unless `system.mutations` is inspected, and the error text points at TTL while the statement the user is waiting on is an ordinary `UPDATE`.

`ReplicatedMergeTree` and `SharedMergeTree` are not affected — their mutations and metadata are Keeper transactions.

**Relationship to #113459.** That issue reports the same commit window — mutation entry durable before the metadata commit — for `ALTER ... RENAME COLUMN`, where the orphan mutation *applies* and silently returns defaults. This is the opposite outcome of the same window: a mutation that can never apply and blocks the queue behind it. I am filing separately because the symptom, the severity axis and the search terms are entirely different, and because a fix scoped to `RENAME COLUMN` would leave this live. If you prefer one issue per root cause, please close this as a duplicate — a single atomicity fix for the pair would cover both.

### Does it reproduce on the most recent release?

Yes, reproduced on `26.8.1.1` (current master build).

### How to reproduce

Reproduced with a SIGKILL delivered between the two commit records; no power loss, no special filesystem and **no non-default settings** are required — verified at production defaults as well as with the fsync family pinned on, with identical results (2 of 6 trials each).

1. Start a plain single-node server.

2. Create a table with **no** TTL and insert some rows:

```sql
CREATE TABLE ttlw_t (id UInt64, val UInt64, ts DateTime) ENGINE = MergeTree ORDER BY id;
INSERT INTO ttlw_t SELECT number, number * 13, now() FROM numbers(300);
```

3. Issue the TTL alter, concurrently with `OPTIMIZE TABLE ttlw_t FINAL` in a loop (the concurrent rename traffic is only there to make the crash land inside the DDL commit window reliably):

```sql
ALTER TABLE ttlw_t MODIFY TTL ts + INTERVAL 3 YEAR;
```

4. `SIGKILL` the server during that window — specifically after `mutation_.txt` has been written and before `.sql.tmp` is renamed onto `.sql`.

5. Restart. The schema has no TTL, the mutation is still there, and it fails on every retry:

```sql
SHOW CREATE TABLE ttlw_t; -- no TTL clause
SELECT command, is_done, latest_fail_reason FROM system.mutations WHERE table = 'ttlw_t';
```

6. Issue any ordinary mutation and observe that it never finishes:

```sql
ALTER TABLE ttlw_t UPDATE val = val + 1 WHERE id < 10;
```

Recovered on-disk state and log lines from one reproduction

The interrupted metadata commit is stated plainly at startup:

```
DatabaseAtomic (default): Removing file store/a10//ttlw_t.sql.tmp
```

The surviving `.sql` is the pre-`ALTER` one, with no TTL:

```
ATTACH TABLE _ UUID '16a02dc2-...'
( `id` UInt64, `val` UInt64, `ts` DateTime )
ENGINE = MergeTree
ORDER BY id
SETTINGS index_granularity = 8192
```

while the table's data directory still holds the orphan entry, and then the blocked follow-up:

```
$ cat store/16a/16a02dc2-.../mutation_7.txt
format version: 1
create time: 2026-08-06 04:54:20
commands: (MATERIALIZE TTL)

$ cat store/16a/16a02dc2-.../mutation_8.txt
format version: 1
create time: 2026-08-06 04:54:20
commands: (UPDATE val = val + 1 WHERE id < 10)
```

The executor retries indefinitely — the same task reappears roughly once a second, here on the part that carries both mutation versions:

```
04:54:20.889 {…::all_1_6_1_7} MutatePlainMergeTreeTask: Code: 80 … Cannot MATERIALIZE TTL as there is no TTL set for table default.ttlw_t
04:54:20.963 default.ttlw_t: Added mutation: mutation_8.txt
04:54:20.963 {…::all_1_6_1_8} MutatePlainMergeTreeTask: Code: 80 … Cannot MATERIALIZE TTL as there is no TTL set for table default.ttlw_t
04:54:21.886 {…::all_1_6_1_8} MutatePlainMergeTreeTask: Code: 80 … (same)
04:54:22.729 {…::all_1_6_1_8} MutatePlainMergeTreeTask: Code: 80 … (same)
```

Data is unaffected throughout: 300 rows and `sum(val) = 583050` before the crash and after recovery. After `KILL MUTATION ... SYNC` both entries disappear and the pending `UPDATE` settles (`sum(val) = 583060`).

### Expected behavior

`ALTER TABLE ... MODIFY TTL` should be all-or-nothing across a crash: either the TTL is in the metadata and the `MATERIALIZE TTL` mutation is queued, or neither is present. A mutation entry should never survive a rollback of the metadata that gives it meaning.

Failing that, a mutation whose command is impossible against the current schema should be discarded — with a log message naming it — rather than retried forever, and it should not prevent unrelated later mutations on the same table from completing.

### Error message and/or stacktrace

```
2026.08.06 04:54:20.889390 [ 3825265 ] {16a02dc2-9d38-429e-8f4d-1baf64737aaf::all_1_6_1_7} MutatePlainMergeTreeTask: Code: 80. DB::Exception: Cannot MATERIALIZE TTL as there is no TTL set for table default.ttlw_t (16a02dc2-9d38-429e-8f4d-1baf64737aaf). (INCORRECT_QUERY) (version 26.8.1.1 (official build))
2026.08.06 04:54:20.889703 [ 3825265 ] {} MergeTreeBackgroundExecutor: Exception while executing background task {16a02dc2-9d38-429e-8f4d-1baf64737aaf::all_1_6_1_7}: Code: 80. DB::Exception: Cannot MATERIALIZE TTL as there is no TTL set for table default.ttlw_t (16a02dc2-9d38-429e-8f4d-1baf64737aaf). (INCORRECT_QUERY)
```

### Additional context

Found by a crash-durability test framework and then reduced to a standalone deterministic reproducer, run outside the framework's own judging path (plain SIGKILL through an `LD_PRELOAD` shim, no fault simulator, no oracles) so that a reproduction cannot be an artifact of the harness. 2 of 6 trials reproduce, at production defaults and with the fsync family pinned alike; the other 4 land outside the window and the orphan mutation completes harmlessly.

Related, same family, both open: #113459 (the same commit window with `RENAME COLUMN`, opposite outcome) and #111380 (the mutation entry file's own lifecycle is not crash-durable — notably, an acknowledged `KILL MUTATION`, the recovery action for this issue, can itself silently revert after power loss).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.