ClickHouse / ClickHouse/ClickHouse
`make_distributed_plan` fails on full-text search queries when direct read from text index is enabled
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
Full-text search over a `text` index works with `make_distributed_plan=1` as long as the index is only used for granule skipping: the filter DAG is serialized and every worker re-runs index analysis over its assigned parts.
However `query_plan_direct_read_from_text_index` is enabled by default, and the direct-read optimization
https://github.com/ClickHouse/ClickHouse/blob/9baaf47a6ec4c6035d8502409dbf3215075d1b42/src/Processors/QueryPlan/Optimizations/optimizeTree.cpp#L226 rewrites eligible text-search functions to read a virtual column produced by the index (`__text_index___`). `tryMakeDistributedRead` then buckets every sufficiently large read without checking for this rewrite: https://github.com/ClickHouse/ClickHouse/blob/9baaf47a6ec4c6035d8502409dbf3215075d1b42/src/Processors/QueryPlan/Optimizations/makeDistributed.cpp#L532-L565.
Note there is already a guard for direct text index tasks in `ReadFromMergeTree` serde that throws a clean `SUPPORT_IS_DISABLED`: https://github.com/ClickHouse/ClickHouse/blob/9baaf47a6ec4c6035d8502409dbf3215075d1b42/src/Processors/QueryPlan/ReadFromMergeTree.cpp#L5047 - but the virtual-column rewrite slips past it.
Repro:
```sql
SET allow_experimental_full_text_index = 1;
CREATE TABLE t_text_repro (id UInt64, s String, INDEX idx_text s TYPE text(tokenizer = splitByNonAlpha)) ENGINE = MergeTree ORDER BY id;
INSERT INTO t_text_repro SELECT number, 'word' || toString(number) FROM numbers(100000);
SET make_distributed_plan = 1, distributed_plan_execute_locally = 1, enable_parallel_replicas = 0,
distributed_plan_max_rows_to_broadcast = 0, distributed_plan_default_reader_bucket_count = 3, max_rows_to_group_by = 0;
SELECT count() FROM t_text_repro WHERE hasAnyTokens(s, ['word42']); -- NOT_FOUND_COLUMN_IN_BLOCK
SELECT count() FROM t_text_repro WHERE hasAnyTokens(s, ['word42'])
SETTINGS query_plan_direct_read_from_text_index = 0; -- works, returns 1
```
So the task, in two steps:
1. Fallback first. Make `tryMakeDistributedRead` decline bucketing when the read carries direct text index state (index read tasks or the virtual-column rewrite), the same way it already declines for FINAL with specialized merging engines: https://github.com/ClickHouse/ClickHouse/blob/9baaf47a6ec4c6035d8502409dbf3215075d1b42/src/Processors/QueryPlan/Optimizations/makeDistributed.cpp#L550-L556. Alternatively the direct-read rewrite could skip reads that will be distributed. Either way the query should fall back to a serial read instead of failing.
2. Support it properly. A non-bucketed read is rebuilt and re-optimized on the worker, which re-derives the direct read - so the natural fix for bucketed reads is to let the worker re-run the text-index rewrite over its pinned part list instead of serializing the rewritten DAG (or, alternatively, serialize the index read tasks per bucket).
Add a test with the repro above once step 1 lands: the query must return identical results with make_distributed_plan = 0 and = 1.
Contributor guide
Assessment
This issue has not been assessed yet.