ClickHouse / ClickHouse/ClickHouse
Parallel replicas over `Merge` tables: derive eligibility and freshness checks from the `_table`/`_database`-filtered child set
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
🕵 With `parallel_replicas_allow_merge_tables`, both the parallel-replicas eligibility check of a `Merge` table (`StorageMerge::supportsParallelReplicasReading`) and the freshness snapshot of its underlying replicated tables (`StorageMerge::getReplicatedChildTableNames`) walk **every** table the `Merge` regexp matches. The read itself later prunes children with the `_database` / `_table` virtual columns (`ReadFromMerge::getSelectedTables`), so the two conservative checks can be wider than the read:
- `SELECT ... FROM m WHERE _table = 'good'` loses parallel replicas when some *other* matched child is `Log` / non-replicated, even though only `good` is read.
- With `max_replica_delay_for_distributed_queries > 0` and `fallback_to_stale_replicas_for_distributed_queries = 0`, a replica lagging only on an *unselected* replicated child is excluded from coordinated reading of the filtered query.
Both are missed optimizations, never correctness issues: the query falls back to non-coordinated reading (or fewer replicas) and returns correct results. The test `04827_parallel_replicas_merge_tables_virtual_column_filter` pins the current behavior.
Deriving both checks from the filtered child set is a protocol-level change, not a local fix:
- Eligibility is decided before the pushed-down filters exist: `findQueryForParallelReplicas` / `findTableForParallelReplicas` and `collectFiltersForAnalysis` are sibling arguments of the `GlobalPlannerContext` constructor (`src/Planner/Planner.cpp`), so the designation walk cannot see the filter DAG today.
- The same eligibility verdict is derived independently at several places — the initiator designation (`src/Planner/findParallelReplicasQuery.cpp`), the follower re-planning of the shipped query, the serialized-plan path (`resolveStorages`), and `PlannerJoinTree` — and the feature's fail-closed guards (designated-table mismatch, and the execution-time revalidation in `ReadFromMerge::filterTablesAndCreateChildrenPlans`) throw `SUPPORT_IS_DISABLED` when the nodes disagree. Narrowing must therefore produce an **identical or stronger** filtered child set on every node; a node whose extraction comes out weaker (e.g. the filter was not pushed down there, `query_plan_filter_push_down = 0`, or the rewritten query resolves differently) would turn today's silent fallback into a query failure.
- Robustness against that asymmetry needs the initiator to ship not only the freshness-checked children (the existing internal setting `parallel_replicas_freshness_checked_tables`) but also the children *proven excluded* by the filter, so a replica that selects a wider set can distinguish "excluded at planning, rows cannot pass the `WHERE`" (benign) from "started matching after planning" (must fail closed).
The infrastructure to reuse is `collectFiltersForAnalysis` (the dummy-plan filter collection that already serves shard skipping and the parallel-replicas row estimate) plus the `_database`/`_table` filter derivation of `ReadFromMerge::getSelectedTables` (`VirtualColumnUtils::splitFilterDagForAllowedInputs` over the two virtual columns).
Related: https://github.com/ClickHouse/ClickHouse/pull/110972
Contributor guide
Research direction
Start with StorageMerge::supportsParallelReplicasReading, StorageMerge::getReplicatedChildTableNames, and ReadFromMerge::getSelectedTables, then trace collectFiltersForAnalysis and the GlobalPlannerContext construction in src/Planner/Planner.cpp. Follow the eligibility decisions in src/Planner/findParallelReplicasQuery.cpp, resolveStorages, and PlannerJoinTree, using test 04827_parallel_replicas_merge_tables_virtual_column_filter as the baseline. Done means filtered child selection remains consistent across planning and execution paths without weakening the fail-closed guards.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100