ClickHouse / ClickHouse/ClickHouse
Interrupted local→S3 TTL move leaves an orphan part → infinite move-retry loop and unbounded S3 PutObject (Code 84 DIRECTORY_ALREADY_EXISTS at store root, zero-copy OFF)
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
## Summary
On a ReplicatedMergeTree with tiered storage (a local volume plus an S3-backed volume) and `allow_remote_fs_zero_copy_replication=0`, a background TTL move (local → S3) interrupted by a process restart at a specific window leaves the part on two disks. After restart, part loading silently drops the duplicate and leaves an **orphan directory at the S3 volume's store path**. The move scheduler then retries the same part forever — each attempt re-uploads the whole part to S3 (PutObject) and fails at the final rename with `Code: 84 DIRECTORY_ALREADY_EXISTS`. With no backoff on the non-zero-copy move path, it loops every ~2s and produces unbounded S3 request cost (we observed ~tens of millions of PutObject / ~tens of TB re-uploaded on a single replica over ~2 days).
## Environment
- **ClickHouse 26.3.10.60** (revision 54517, git hash `6a6d2d137dfc14972ab1f77412a9e2ada5e0698f`). Source line references below were verified against current `master`.
- ReplicatedMergeTree; storage policy with a local volume `vol_ebs` and an S3-backed volume `vol_s3` (`prefer_not_to_merge=1` on `vol_s3`); a `TTL ... TO VOLUME 'vol_s3'` rule moves aged parts from local to S3.
- `allow_remote_fs_zero_copy_replication=0`.
- Frequent forced restarts (k8s node rotation that recreates the local PVC) while moves to `vol_s3` are active.
## Mechanism (source-traced)
1. `MergeTreePartsMover::clonePart` copies the part to the destination disk's `moving/` dir (one PutObject per file on S3), then `renameTo` moves it into the `vol_s3` **store** path `store///`, then `swapActivePart` commits it and drops the source. The rename→swap window is explicitly flagged at `MergeTreePartsMover.cpp:366-369`.
2. If the process is killed after the rename but before swap, the part exists on **both** the local source disk and `vol_s3`.
3. On restart, `PartLoadingTree::add` (`MergeTreeData.cpp:1692-1739`) keys parts by `MergeTreePartInfo` in a `std::map`; two same-info copies on different disks collapse to a single node (`emplace` no-ops the duplicate key, `:1737`). The S3 copy is dropped before instantiation, so the checksum-based duplicate cleanup (`:2006-2012`, `:2346-2350`) never runs on it — leaving an **orphan** directory at the `vol_s3` store path that is never removed.
4. The move scheduler re-selects the (local) active part for `vol_s3`. `clonePart` re-uploads the entire part to S3 again, then `renameTo` collides with the orphan and throws `Code: 84 DIRECTORY_ALREADY_EXISTS`. The guard flag is hard-coded off: `part.renameTo(part.name, /* remove_new_dir_if_exists */ false)` (`MergeTreePartsMover.cpp:313`).
5. Because the rename failed, `swapActivePart` is never reached and the part stays active. The non-zero-copy move path has **no backoff** (the backoff at `MergeTreeData.cpp:9498` only applies when zero-copy is ON), so the scheduler retries the same part every ~2s indefinitely; each retry re-uploads the full part.
Full stack trace (the same part repeats every ~2s; table uuid / part name / disk name anonymized, addresses are from our build — use the git hash above to map):
```
MergeTreeBackgroundExecutor: Exception while executing background task {::lambda}: Code: 84. DB::Exception: Part directory /var/lib/clickhouse/disks//store/// already exists. (DIRECTORY_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000141cccd0
1. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000e06f598
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000e06ee8c
3. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&&) @ 0x000000000e06ead8
5. DB::IMergeTreeDataPart::renameTo(String const&, bool) @ 0x0000000019dcd490
7. DB::MergeTreeData::moveParts(std::shared_ptr const&, DB::ReadSettings const&, DB::WriteSettings const&, bool) @ 0x0000000019f6016c
8. bool std::__function::__policy_func::__call_func[abi:fe210105](std::__function::__policy_storage const*) @ 0x0000000019f965bc
```
## Why this is not a duplicate
Prior Code 84 reports (#58540, #62830) collide at the `moving/` **temp** dir and were closed without a code fix (recurrences were reported afterward). This one differs in three ways:
- the collision is at the destination **store root** `store///`, against an orphan from an interrupted move — not a stale `moving/` temp dir, so the usual "delete the `moving/` dir" workaround does not apply;
- the orphan originates from `PartLoadingTree::add` silently dropping a cross-disk same-`MergeTreePartInfo` duplicate (a load-tree dedup gap), which has not been diagnosed in any prior issue;
- the impact is unbounded S3 **request cost** (clone-before-rename + no backoff), which no prior issue frames.
## Impact
Correctness: an endless `DIRECTORY_ALREADY_EXISTS` loop; the part never moves to S3. Cost: each retry re-uploads the entire part to S3. On one replica we observed ~32M PutObject and ~18 TB re-uploaded over ~2 days, with bucket-wide PutObject spiking ~100x baseline despite no corresponding new ingested data.
## Proposed fixes (any one breaks the loop)
1. **Load-tree**: when `PartLoadingTree::add` sees the same `MergeTreePartInfo` on a different disk, load both and let the duplicate cleanup remove the loser instead of silently dropping it (so no orphan is left).
2. **Move backoff**: add backoff / repeated-failure skip to the non-zero-copy move path (parity with the zero-copy branch at `:9498`).
3. **Preflight**: check the destination final dir before `clonePart`, so a guaranteed-to-fail move does not re-upload the whole part first (caps the cost even if the loop persists).
## What we directly observed (production, 26.3.10.60)
After a forced restart, the same part directory repeatedly fails with `Code: 84 DIRECTORY_ALREADY_EXISTS` at the `vol_s3` store path every ~2s for hours, while bucket S3 PutObject stays ~100x baseline with no corresponding new active data. The exception stack is consistently `scheduleDataMovingJob → moveParts → renameTo`, and the orphan part directory exists at the `vol_s3` store path while the same part is active on the local disk. (`part_log`/`metric_log` were disabled on this cluster, so per-part attribution comes from the server log plus `system.events` counters rather than `part_log`.)
## Hypothesized minimal reproduction (NOT yet executed)
We have **not** run this minimal repro — it is derived from the source paths above and may need adjustment. The mechanism itself is what we observed in production (above); this is our best guess at the smallest deterministic trigger:
1. Tiered policy (local `vol_ebs` → S3-backed `vol_s3`, zero-copy off), a table with `TTL ... TO VOLUME 'vol_s3'`.
2. While a move to `vol_s3` is in flight, kill the server after the rename into `vol_s3` but before `swapActivePart` — or directly construct the post-condition: leave a part directory at the `vol_s3` store path `store///` while the same part remains active on `vol_ebs`.
3. Restart and observe: the orphan at the `vol_s3` store path, repeated `Code: 84` every ~2s, and continuous S3 PutObject with no net data growth.
Happy to provide more detail or test a candidate fix.
Contributor guide
Assessment
This issue has not been assessed yet.