ClickHouse / ClickHouse/ClickHouse
Iceberg with iceberg_use_version_hint=1: remove_orphan_files deletes an acked snapshot's data behind a stale version-hint (permanent data loss)
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Describe what's wrong
On an Iceberg table with `iceberg_use_version_hint = 1`, `ALTER TABLE ... EXECUTE remove_orphan_files` can **permanently delete the data of a committed, acknowledged snapshot** whenever the table's `version-hint.text` has fallen behind the real latest metadata. The `INSERT` returns success, its rows are durable in the object store, and then a routine orphan-file cleanup erases them.
This is the permanent-data-loss escalation of the recoverable write-wedge in #114108. Both share one root cause: the `version-hint.text` advance is best-effort and its failure is swallowed.
`writeMetadataFileAndVersionHint` (`src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp`) does, in order:
1. a conditional PUT of `vN.metadata.json` with `write-if-none-match: *` — **this is the commit** (`Utils.cpp:342-349`);
2. a separate loop that advances `version-hint.text` to `N`, whose write failures are caught and swallowed, and the function returns `true` regardless (`Utils.cpp:417-420`, `:429`).
So any transient failure of *just* the hint PUT (an S3 throttle / 5xx / connection reset, or the crash window of #114108) leaves `vN.metadata.json` and all of snapshot `N`'s manifests and data files durable while `version-hint.text` still points at `N-1` — and the `INSERT` is acknowledged.
`remove_orphan_files` then trusts that stale hint. `collectReachableFiles` resolves the "current" metadata through `getLatestOrExplicitMetadataFileAndVersion` with `ignore_explicit_metadata_file_path = true` (`src/Storages/ObjectStorage/DataLakes/Iceberg/SnapshotFilesTraversal.cpp:132-142`). That call still takes the `iceberg_use_version_hint` branch (`Utils.cpp:1320-1335`), so reachability is rooted at `N-1`. Snapshot `N`'s `vN.metadata.json`, its manifests and its data files are all unreachable from `N-1` and are classified as orphans.
The only guard left is the age window: `findOrphanFiles` spares a file whose `last_modified >= older_than` (`src/Storages/ObjectStorage/DataLakes/Iceberg/RemoveOrphanFilesExecute.cpp:143`). But the hint divergence is **permanent** (nothing ever re-advances the hint on its own), so once snapshot `N` is older than `older_than` — which happens by default after `iceberg_orphan_files_older_than_seconds = 259200` (3 days), or immediately with `older_than = now()` — every object of the acked snapshot `N` is deleted, including `vN.metadata.json`. The acked rows are then gone from every read path.
The metadata-version recheck (`RemoveOrphanFilesExecute.cpp:280-286`) does not help: it re-runs the same stale-hint resolution, so it compares `N-1` to `N-1` and passes.
### Does it reproduce on the most recent release?
Yes — reproduced deterministically (5/5) on `26.8.1.1`; the code paths above are present on current `master` and are identical in the open-source and ClickHouse Cloud builds.
### How to reproduce
`26.8.1.1`, single node, Iceberg over S3/MinIO, with an S3 proxy between ClickHouse and the object store so that just the `version-hint.text` PUT can be failed while the `vN.metadata.json` and data PUTs go through.
```sql
CREATE TABLE t (x Int64) ENGINE = IcebergS3('http://.../warehouse/t/', 'key', 'secret')
SETTINGS iceberg_use_version_hint = 1;
SET allow_experimental_database_iceberg = 1, allow_insert_into_iceberg = 1, write_full_path_in_iceberg_metadata = 1;
INSERT INTO t SELECT number FROM numbers(20); -- commits v1, hint -> v1, 20 rows
-- Fail ONLY the version-hint.text PUT of the next INSERT (e.g. a proxy returns
-- 403 AccessDenied, which ClickHouse treats as unretryable, on PUTs whose path
-- contains "version-hint.text"). The vN.metadata.json PUT and the data files
-- land normally, the hint advance is swallowed, and the INSERT is ACKNOWLEDGED:
INSERT INTO t SELECT number FROM numbers(20, 20); -- acked; hint still at v1
-- Snapshot v2 (the 20 acked rows) is durable on disk but hidden by the stale hint.
-- A routine orphan cleanup now runs (the default age window is 3 days; used here
-- with older_than = now() to reach the same state immediately):
SET allow_iceberg_remove_orphan_files = 1;
ALTER TABLE t EXECUTE remove_orphan_files(older_than = 'YYYY-MM-DD HH:MM:SS');
-- ^ every data/manifest object of the acked snapshot v2 is deleted.
```
Observed with a direct object-store listing across 5/5 runs: before `remove_orphan_files` the acked snapshot's data/manifest objects are present; after it they are gone (the seed snapshot `v1`, reachable from the stale hint, is spared — so the command is behaving as designed, it just trusts the wrong pointer). A control run on a table whose hint is *not* stale deletes a planted true orphan and spares every reachable file, confirming the command is discriminating and the loss is caused specifically by the stale-hint reachability root.
### Expected behaviour
`remove_orphan_files` must not delete files belonging to a committed snapshot. Reachability should be computed from the true latest metadata (re-derived by listing, independent of `version-hint.text`), or the command should refuse to run / fail loud when it detects that `version-hint.text` lags the newest `vN.metadata.json` in the store, rather than treating the newer committed snapshot as garbage. (Relatedly, the swallowed `version-hint.text` advance in #114108 leaves the table in exactly this divergent state.)
Contributor guide
Assessment
This issue has not been assessed yet.