ClickHouse / ClickHouse/ClickHouse

ReplicatedMergeTree: CHECK TABLE fails after fetching part with unknown projection (re-fetch loop)

Open
#100,413 6 comments 0 reactions 1 assignee Assigned to @CheSema View on GitHub
bug comp-projections comp-replication
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Describe what's wrong

PR #99623 fixed parts with unknown projections being marked as lost, but the fix is **incomplete on the receiver side** (replica that fetches the part). After a successful fetch, a subsequent `CHECK TABLE` or the background part-check thread calls `checkDataPart` and fails with `NO_FILE_IN_DATA_PART`, marking the part as broken and triggering an infinite re-fetch loop.

**Root cause:**

`downloadPartToDisk` (fixed by #99623) patches `data_checksums` in memory so the immediate `checkEqual` passes — but `checksums.txt` is still written to disk verbatim from the sender, including the `pp.proj` entry that was never transferred (because the projection is unknown, `getProjectionParts()` returns 0 for it so `send_projections` loop skips it).

When `checkDataPart` later runs on replica 2:

- `projections_on_disk` is built by iterating the part directory for `.proj` subdirectories — `pp.proj` does **not** exist on disk, so `projections_on_disk` stays **empty** for it.
- `getProjectionParts()` does not include `pp` (unknown projection).
- The new guard added by #99623 — `if (!projections_on_disk.empty())` — **does not fire**.
- `checksums_txt` (loaded from disk) still has `pp.proj`.
- `checksums_data` does not have `pp.proj` (directory never existed on disk).
- `checksums_txt.checkEqual(checksums_data)` throws `NO_FILE_IN_DATA_PART`.

The part is marked broken → re-fetched from replica 1 → same outcome → **infinite loop**.

The fix in `checkDataPart.cpp` only handles the **local-attach case** (`.proj` directory exists on disk but is unknown to the table, i.e. `projections_on_disk` is non-empty). It does **not** handle the **fetch case** (`checksums.txt` references a `.proj` entry whose directory was never transferred).

### Does it reproduce on the most recent release?

Yes — introduced/unblocked by #99623 (merged 2026-03-21).

### How to reproduce

Requires ReplicatedMergeTree with 2 replicas and a ZooKeeper/Keeper.

```sql
-- On both replicas:
CREATE TABLE t (x Int32, y Int32, PROJECTION p (SELECT x, y ORDER BY x))
ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/t', '{replica}')
PARTITION BY intDiv(y, 100) ORDER BY y
SETTINGS max_parts_to_merge_at_once = 1;

-- On replica 1:
INSERT INTO t SELECT number, number FROM numbers(7);
ALTER TABLE t ADD PROJECTION pp (SELECT x, count() GROUP BY x);
ALTER TABLE t MATERIALIZE PROJECTION pp SETTINGS mutations_sync=2;
ALTER TABLE t DETACH PARTITION 0;

-- On replica 2: remove the detached copy so it is forced to re-fetch later
ALTER TABLE t DROP DETACHED PARTITION 0 SETTINGS allow_drop_detached=1;

-- On replica 1: drop projection while part is detached, then re-attach
ALTER TABLE t DROP PROJECTION pp;
ALTER TABLE t ATTACH PARTITION 0;

-- On replica 2: sync, then run CHECK TABLE
SYSTEM SYNC REPLICA t;
SELECT count() FROM t; -- returns 7 (data is fine)
CHECK TABLE t; -- fails: NO_FILE_IN_DATA_PART for pp.proj
```

The `SELECT` succeeds because data is intact. `CHECK TABLE` fails because `checksums.txt` references `pp.proj` but the directory was never transferred. The background `ReplicatedMergeTreePartCheckThread` will then mark the part as broken and re-fetch it from replica 1, creating a loop.

The same state can be reproduced without replication — manually remove a `.proj` directory from a detached part whose `checksums.txt` still references it, then attach and run `CHECK TABLE`:

```bash
# After the detach step above:
rm -rf /var/lib/clickhouse/data/default/t/detached//pp.proj
```

```sql
ALTER TABLE t ATTACH PARTITION 0;
SELECT count() FROM t; -- 7, works
CHECK TABLE t; -- Code: 226. NO_FILE_IN_DATA_PART: No file pp.proj in data part
```

### Suggested fix

In `checkDataPart.cpp`, after the existing new block (which handles projections whose directories exist on disk), also remove from `checksums_txt` any `.proj` entries that are referenced in `checksums_txt` but absent from `checksums_data` (i.e., whose directory was never on disk):

```cpp
/// Also handle the case where checksums_txt references a .proj entry whose
/// directory is absent from disk entirely (e.g. receiver after an inter-replica
/// fetch of a part with an unknown projection — checksums.txt was transferred
/// but the .proj directory was not).
for (auto it = checksums_txt.files.begin(); it != checksums_txt.files.end(); )
{
if (it->first.ends_with(".proj") && !checksums_data.has(it->first))
{
is_broken_projection = true;
it = checksums_txt.files.erase(it);
}
else
++it;
}
```

The test `03822_attach_with_unknown_projection.sh` should also add `CHECK TABLE t_unknown_proj_2` after `SYSTEM SYNC REPLICA t_unknown_proj_2` to cover this path.

### Related

- #99623 (incomplete fix)
- #95952 (original fix, reverted)
- #95991 (revert of #95952)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.