ClickHouse / ClickHouse/ClickHouse

Parallel replicas over `Merge` tables: derive eligibility and freshness checks from the `_table`/`_database`-filtered child set

Open
#113,985 0 comments 0 reactions 0 assignees View on GitHub
comp-parallel-replicas performance
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

🕵 With `parallel_replicas_allow_merge_tables`, both the parallel-replicas eligibility check of a `Merge` table (`StorageMerge::supportsParallelReplicasReading`) and the freshness snapshot of its underlying replicated tables (`StorageMerge::getReplicatedChildTableNames`) walk **every** table the `Merge` regexp matches. The read itself later prunes children with the `_database` / `_table` virtual columns (`ReadFromMerge::getSelectedTables`), so the two conservative checks can be wider than the read:

- `SELECT ... FROM m WHERE _table = 'good'` loses parallel replicas when some *other* matched child is `Log` / non-replicated, even though only `good` is read.
- With `max_replica_delay_for_distributed_queries > 0` and `fallback_to_stale_replicas_for_distributed_queries = 0`, a replica lagging only on an *unselected* replicated child is excluded from coordinated reading of the filtered query.

Both are missed optimizations, never correctness issues: the query falls back to non-coordinated reading (or fewer replicas) and returns correct results. The test `04827_parallel_replicas_merge_tables_virtual_column_filter` pins the current behavior.

Deriving both checks from the filtered child set is a protocol-level change, not a local fix:

- Eligibility is decided before the pushed-down filters exist: `findQueryForParallelReplicas` / `findTableForParallelReplicas` and `collectFiltersForAnalysis` are sibling arguments of the `GlobalPlannerContext` constructor (`src/Planner/Planner.cpp`), so the designation walk cannot see the filter DAG today.
- The same eligibility verdict is derived independently at several places — the initiator designation (`src/Planner/findParallelReplicasQuery.cpp`), the follower re-planning of the shipped query, the serialized-plan path (`resolveStorages`), and `PlannerJoinTree` — and the feature's fail-closed guards (designated-table mismatch, and the execution-time revalidation in `ReadFromMerge::filterTablesAndCreateChildrenPlans`) throw `SUPPORT_IS_DISABLED` when the nodes disagree. Narrowing must therefore produce an **identical or stronger** filtered child set on every node; a node whose extraction comes out weaker (e.g. the filter was not pushed down there, `query_plan_filter_push_down = 0`, or the rewritten query resolves differently) would turn today's silent fallback into a query failure.
- Robustness against that asymmetry needs the initiator to ship not only the freshness-checked children (the existing internal setting `parallel_replicas_freshness_checked_tables`) but also the children *proven excluded* by the filter, so a replica that selects a wider set can distinguish "excluded at planning, rows cannot pass the `WHERE`" (benign) from "started matching after planning" (must fail closed).

The infrastructure to reuse is `collectFiltersForAnalysis` (the dummy-plan filter collection that already serves shard skipping and the parallel-replicas row estimate) plus the `_database`/`_table` filter derivation of `ReadFromMerge::getSelectedTables` (`VirtualColumnUtils::splitFilterDagForAllowedInputs` over the two virtual columns).

Related: https://github.com/ClickHouse/ClickHouse/pull/110972

Contributor guide

Open the contributing guide

Research direction

Start with StorageMerge::supportsParallelReplicasReading, StorageMerge::getReplicatedChildTableNames, and ReadFromMerge::getSelectedTables, then trace collectFiltersForAnalysis and the GlobalPlannerContext construction in src/Planner/Planner.cpp. Follow the eligibility decisions in src/Planner/findParallelReplicasQuery.cpp, resolveStorages, and PlannerJoinTree, using test 04827_parallel_replicas_merge_tables_virtual_column_filter as the baseline. Done means filtered child selection remains consistent across planning and execution paths without weakening the fail-closed guards.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.