ClickHouse / ClickHouse/ClickHouse
Logical error: 'Coordinator for parallel reading from replicas is not initialized' for cluster() inside an IN subquery with parallel_replicas_local_plan = 0
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Describe the bug
`Logical error: 'Coordinator for parallel reading from replicas is not initialized'` is thrown at
`src/QueryPipeline/RemoteQueryExecutor.cpp:877` in
`RemoteQueryExecutor::processMergeTreeInitialReadAnnouncement` when a `cluster()` read sits inside an
`IN` subquery of a query that reads a MergeTree table with parallel replicas, and
`parallel_replicas_local_plan = 0`.
The remote side sends `Protocol::Server::MergeTreeAllRangesAnnouncement`, but the initiator's
`RemoteQueryExecutor` for that nested read has no `extension->parallel_reading_coordinator`, so the
guard aborts. There is an identical guard at `:867` in `processMergeTreeReadTaskRequest`.
This is not issue #77990 (closed 2025-12-03). That report's own reproducer, a `JOIN cluster(...)`,
passes on current master with `parallel_replicas_local_plan` set to either 0 or 1. The `IN`-subquery
carrier below is a different, still-live path.
### How to reproduce
Version `26.8.1.1` (master, debug build). A single server is enough: define a two-replica cluster
whose replicas both point at that server.
```xml
127.0.0.19000
127.0.0.29000
```
```sql
CREATE TABLE null_in_pr (dt DateTime, idx Int32, i Nullable(UInt64))
ENGINE = MergeTree PARTITION BY dt ORDER BY idx;
INSERT INTO null_in_pr SELECT toDateTime(number), number, number FROM numbers(100);
SELECT count() FROM null_in_pr AS t
WHERE t.i IN (SELECT i FROM cluster('test_cluster_one_shard_two_replicas', currentDatabase(), 'null_in_pr'))
SETTINGS allow_experimental_parallel_reading_from_replicas = 1, max_parallel_replicas = 3,
cluster_for_parallel_replicas = 'test_cluster_one_shard_two_replicas',
parallel_replicas_for_non_replicated_merge_tree = 1,
parallel_replicas_local_plan = 0;
```
Deterministic: 3 out of 3 runs, each on a fresh server and a fresh data directory.
### What is load-bearing
All three conditions are required. Each arm below is a fresh server; the same query text differs only
as stated.
| Query shape (`parallel_replicas_local_plan = 0` throughout) | Result |
| --- | --- |
| `cluster()` inside an `IN` subquery, outer table read with parallel replicas | logical error |
| bare `SELECT count() FROM cluster(...)` | 100 |
| local MergeTree read only, no `cluster()` | 100 |
| `cluster()` in a scalar subquery | 200 |
| `cluster()` in a `FROM` subquery | 100 |
| `IN` subquery over a local table, no `cluster()` | 100 |
And the setting arms, with a negative control:
| Settings | Result |
| --- | --- |
| parallel replicas on, `parallel_replicas_local_plan = 0` | logical error |
| parallel replicas on, `parallel_replicas_local_plan = 1` | 100 |
| parallel replicas on, `compatibility = '24.3'` | logical error |
| parallel replicas off entirely | 100 |
`compatibility = '24.3'` reproduces only because it flips the setting. Measured on the same server:
`parallel_replicas_local_plan` is 1 by default, 0 under `compatibility = '24.3'`, and 1 again under
`compatibility = '24.11'`, matching the `false -> true` default change recorded in the 24.10 and
24.11 blocks of `SettingsChangesHistory.cpp`.
That matters for exposure: `tests/clickhouse-test:1602` randomizes `parallel_replicas_local_plan`
between 0 and 1 on every functional-test run, so this is reachable well beyond the one AST fuzzer job
that pins `compatibility` to `24.3` (`ci/jobs/ast_fuzzer_job.py:359-366`).
### CI sightings
Both on 2026-08-03, both on PRs that do not touch the coordinator or `RemoteQueryExecutor`, 0 rows on
master:
- `Stress test (amd_msan)`, STID `6383-2785`: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=112360&sha=13b616e604e165ad81e458af2418c564bdc55900&name_0=PR&name_1=Stress%20test%20%28amd_msan%29 - the query there is the `03232_pr_not_ready_set.sql` shape, `ignore(...) IN (SELECT query_id FROM cluster(...))`
- `AST fuzzer (amd_debug, targeted, old_compatibility)`, STID `6383-451e`: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=102192&sha=0333e0622c5db34c20ba2375823da23f1e44e4c9&name_0=PR&name_1=AST%20fuzzer%20%28amd_debug%2C%20targeted%2C%20old_compatibility%29
Same guard, same nesting, so the two STIDs are one defect.
### Error message and/or stacktrace
```
: Logical error: 'Coordinator for parallel reading from replicas is not initialized'.
4. src/Common/Exception.h:219:77: DB::Exception::Exception<>(int, FormatStringHelperImpl<>)
5. src/QueryPipeline/RemoteQueryExecutor.cpp:877:15: DB::RemoteQueryExecutor::processMergeTreeInitialReadAnnouncement(DB::InitialAllRangesAnnouncement)
6. src/QueryPipeline/RemoteQueryExecutor.cpp:718:13: DB::RemoteQueryExecutor::processPacket(DB::Packet)
7. src/QueryPipeline/RemoteQueryExecutor.cpp:698:28: DB::RemoteQueryExecutor::readAsync()
8. src/Processors/Sources/RemoteSource.cpp:189:36: DB::RemoteSource::tryGenerate()
9. src/Processors/ISource.cpp:119:26: DB::ISource::work()
10. src/Processors/Executors/ExecutionThreadContext.cpp:127:28: DB::ExecutionThreadContext::executeTask()
11. src/Processors/Executors/PipelineExecutor.cpp:384:26: DB::PipelineExecutor::executeStepImpl(unsigned long, DB::WorkloadResources&&, std::atomic*)
```
In the `Stress test (amd_msan)` sighting two follower threads abort simultaneously under one
`initial_query_id`, which is consistent with the nested read being started per follower rather than
with a per-connection race.
### Additional context
The guard itself looks correct: an executor with no coordinator cannot answer an announcement, and
downgrading the throw to a warning would silently drop the announcement. The defect is upstream,
either the announcement being routed to a connection belonging to a nested non-parallel-replicas
read, or the nested `cluster()` read inheriting a parallel-replicas context it should not. I am
looking into it and will link a fix here.
Contributor guide
Research direction
Reproduce the query with parallel_replicas_local_plan = 0, then inspect src/QueryPipeline/RemoteQueryExecutor.cpp at the guards around lines 867 and 877. Trace how the nested cluster() read and its parallel-reading coordinator are initialized, using tests/clickhouse-test and the cited 03232_pr_not_ready_set.sql shape as entry points. Done means the reproducer returns successfully without weakening the guards and has regression coverage for the IN-subquery case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100