Investigation: stateless / Scraping part_log.2 FAIL — replica crash "Temporary part ... already added" on DatabaseReplicated (stable-25.8)

Affected tests (meta-failures):

• Scraping part_log.2 (the visible symptom in the CI report)
• Killed by signal (in clickhouse-server.log or clickhouse-server.err.log)
• Logical error thrown (in clickhouse-server.log or clickhouse-server.err.log)
• Fatal messages (in clickhouse-server.log or clickhouse-server.err.log)

All four are downstream symptoms of the same underlying server abort.

Affected jobs / branches:

• Stateless tests (amd_binary, old analyzer, s3 storage, DatabaseReplicated, sequential) — 2 Scraping part_log.2 FAILs in 185 runs over 90 days (~1.1% overall; 2/24 = ~8.3% on stable-25.8 specifically). First seen 2026-04-08, last 2026-07-07.
• Same job on upstream master (issue ClickHouse/ClickHouse#86050) — closed as flaky in Feb/2026.

Sample failing run:

• Report: https://s3.amazonaws.com/altinity-build-artifacts/REFs/stable-25.8/2bf2745db0b97132ef23be46b39da714774d383e/28667030517/ci_run_report.html#checks-fails
• GH job: https://github.com/Altinity/ClickHouse/actions/runs/28667030517/job/85033048808
• stable-25.8 @ SHA 2bf2745d

Description

When it hits, clickhouse-server1 aborts (SIGABRT, signal 6) mid-run during a background replica fetch. Killing one replica cascades into DDL-queue stalls, 160 s timeouts on subsequent tests, and finally the meta-check Scraping part_log.2 FAILs because the framework can no longer collect part_log from the degraded cluster. Scraping part_log.2 is the most visible symptom in the CI report, but the underlying cause is the replica crash.

Rare: 2 occurrences in 185 runs of this job in 90 days, both on stable-25.8. Not tied to any Altinity-specific change — pattern predates PR #1875 (first observed 2026-04-08).

Root Cause

LOGICAL_ERROR in TemporaryParts::add when the same clone target name is registered twice during a replica fetch:

<Fatal> : Logical error: 'Temporary part tmp_clone_all_85_85_0_86 already added'.
...
 5. src/Storages/MergeTree/TemporaryParts.cpp:23   TemporaryParts::add(String const&)
 6. src/Storages/MergeTree/MergeTreeData.cpp:3050  MergeTreeData::getTemporaryPartDirectoryHolder(...)
 7. src/Storages/MergeTree/MergeTreeData.cpp:8480  MergeTreeData::cloneAndLoadDataPart(...)
15. src/Storages/StorageReplicatedMergeTree.cpp:5308  fetchPart lambda
16. src/Storages/StorageReplicatedMergeTree.cpp:5360  StorageReplicatedMergeTree::fetchPart(...)
17. src/Storages/StorageReplicatedMergeTree.cpp:2602  StorageReplicatedMergeTree::executeFetch(...)

Two concurrent fetchPart attempts on the same part name (tmp_clone_all_85_85_0_86) hit TemporaryParts::add — the second call throws LOGICAL_ERROR, which triggers abortOnFailedAssertion() in release builds and kills the server.

Upstream has a related fix (ClickHouse/ClickHouse#91792, merged Dec/2025) that plugs the analogous race in the local merge/mutate path (Merge/MutatePlainMergeTreeTask::cancel()), but does not touch the fetchPart path. Not backported to stable-25.8 (also has no upstream backport PR). A more recent open PR ClickHouse/ClickHouse#108879 proposes reclaiming stale empty-part directories instead of throwing — may cover this path but still under review.

Suggested Next Steps

1. File an upstream issue referencing this stack trace, #86050, and #91792 — request that the same tmp-dir race fix be applied to StorageReplicatedMergeTree::fetchPart / MergeTreeData::cloneAndLoadDataPart.
2. Monitor ClickHouse/ClickHouse#108879 — if merged, evaluate backport.
3. Optional: mark as broken in tests/broken_tests.yaml — if the rate climbs on stable-25.8 (currently 2/24 = ~8.3%), an entry filtered to check_types: [DatabaseReplicated] and message: 'Temporary part.*already added' (regex) would suppress the four cascading meta-failures (Killed by signal, Logical error thrown, Fatal messages, Scraping part_log.2) without masking unrelated crashes on the same job. For now, rerun on hit is enough given the low frequency.

References

• Upstream flaky-test tracker for the exact same error and same job: ClickHouse/ClickHouse#86050 (closed as flaky test)
• Related upstream CI crash tracker: ClickHouse/ClickHouse#91132
• Partial upstream fix (local merge/mutate only, not fetchPart): ClickHouse/ClickHouse#91792
• Related open upstream fix: ClickHouse/ClickHouse#108879
• Altinity CI data: `gh-data.checks` filtered on `check_name = 'Stateless tests (amd_binary, old analyzer, s3 storage, DatabaseReplicated, sequential)'` and `test_name IN ('Killed by signal ...', 'Logical error thrown ...', 'Fatal messages ...', 'Scraping part_log.2')`

Start with TemporaryParts::add in src/Storages/MergeTree/TemporaryParts.cpp and trace the cited fetch path through MergeTreeData.cpp and StorageReplicatedMergeTree.cpp. Compare the related fixes in ClickHouse#91792 and #108879, then rerun the DatabaseReplicated stateless job or investigate the supplied failing run. Done means the duplicate temporary-part condition no longer aborts the replica and the upstream issue or backport path is documented.