ClickHouse / ClickHouse/ClickHouse

`ALTER TABLE ... DROP DETACHED PART/PARTITION` is not durable: the removal is never fsynced and no setting reaches this path, so the detached part resurrects after power loss

Open
#111,349 3 comments 0 reactions 1 assignee Claimed by @Michicosun View on GitHub
bug comp-mergetree minor
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Describe what's wrong

`ALTER TABLE t DROP DETACHED PARTITION ... SETTINGS allow_drop_detached = 1` acknowledges success, but the removal it performs — rename of `detached/` to `detached/deleting_` followed by a recursive unlink (`MergeTreeData::dropDetached` → `removeDetachedPart`) — issues no fsync and no directory sync. After a power loss within the filesystem-journal commit window (~5 s on default ext4), the directory operations roll back and the "dropped" detached part is back in `detached/`, fully intact, contradicting the acknowledgement.

Unlike `DROP PARTITION` on active parts (which is made durable by empty covering parts when `fsync_after_insert` + `fsync_part_directory` are on), **no setting reaches the detached-parts path**: `fsync_part_directory` is not consulted there. There is currently no way to make an acked `DROP DETACHED PART/PARTITION` durable.

Detached parts are exactly the state users purge for data-removal purposes — a detached part holds real data readable off disk — so a silently-resurrecting one matters.

### Does it reproduce on the most recent release?

Yes (reproduced on `26.7.1.1311`, current master nightly).

### How to reproduce

* Single node, data directory on ext4.

```sql
CREATE TABLE t (id UInt64, p UInt8) ENGINE = MergeTree ORDER BY id
SETTINGS fsync_after_insert = 1, fsync_part_directory = 1;
INSERT INTO t SELECT number, 1 FROM numbers(1000);
ALTER TABLE t DETACH PARTITION tuple();
-- sync(2) here: the detached part is durable on disk
ALTER TABLE t DROP DETACHED PARTITION tuple() SETTINGS allow_drop_detached = 1; -- acknowledged
-- power loss within the journal-commit window, restart
SELECT name FROM system.detached_parts WHERE table = 't'; -- all_1_1_0 (!)
```

Deterministic harness: data directory on ext4 over `dm-flakey` (power cut = suspend `--noflush` + `drop_writes` + remount; journal recovery yields the exact power-cut state). 4/4 deterministic with a wide journal-commit interval (`-o commit=60`); probabilistic within the `commit=` interval (default 5 s) otherwise.

Trust guards, all passing in every run:

* An un-fsynced canary file written just before the `DROP DETACHED` is always lost — the power cut is real.
* Graceful control: `DROP DETACHED` followed by a clean restart never resurrects the part — the resurrection is power-loss-specific, not restart logic.

### Expected behavior

Either the removal should be fsynced (sync the `detached/` directory after the unlinks, at least when the table has `fsync_part_directory = 1`), or the limitation should be documented.

### Error message and/or stacktrace

None — the resurrection is silent; the part simply reappears in `system.detached_parts`.

### Additional context

Sibling of #111348 (the un-fsynced Atomic-database DDL commit records: `DROP TABLE`/`RENAME`/`EXCHANGE` silently revert after power loss) and of #111322 (`DETACH ... PERMANENTLY` flag file never fsynced). Found by the same framework as #111269 / #111318 / #111320 / #111321 / #111330 / #111331 / #111339.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.