ClickHouse / ClickHouse/ClickHouse

ARRAY JOIN / array string-filter executor regressions: Nested +25% in 25.9, all shapes +30–75% in 26.1, Nested ×2 in 26.6

Open
#111,410 8 comments 0 reactions 1 assignee Claimed by @benj-chen View on GitHub
comp-query-execution external performance
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

Company or project name: JetBrains — IJ Perf, the IntelliJ performance dashboard (https://github.com/JetBrains/ij-perf-report-aggregator)

### Describe the situation

While evaluating an upgrade from 24.3 LTS we found that every dashboard chart query regressed
20–55% on 26.3/26.6. These queries unnest a `Nested` column of per-report measurements
(`ARRAY JOIN measures`) and filter on the measurement name. A per-month bisect on a
self-contained synthetic repro (below) shows **three separate steps**:

1. **v25.9** — `ARRAY JOIN` over a `Nested` column got ~25% slower (single-array shapes unaffected).
2. **v26.1** — filtering unnested string array elements got 30–75% slower across every shape we
tested: plain `Array(String)`, `Array(LowCardinality(String))`, `Nested`, and even
`arrayCount(x -> x = c, arr)` with no ARRAY JOIN at all.
3. **v26.6** — `ARRAY JOIN` over `Nested` with `LowCardinality` takes another large step
(~1.9× vs 25.8 overall).

Each step is present in the earliest published release of its series (v25.9.2.1, v26.1.1.912),
so these are not backports into later patches. (v26.1.1.912 was in fact worse — 1238 ms on q4
below vs 1039 ms on v26.1.12.23 — later 26.1 patches recovered part of it.)

The loss is executor CPU, not I/O or planning: `read_rows`, selected marks, and read bytes are
identical across versions, and single-threaded ratios match multi-threaded ones.

### Which ClickHouse versions are affected?

`report` is our production schema: `measures` is
`Nested(name LowCardinality(String), value Int32, type LowCardinality(String))` on a MergeTree
with a handful of LowCardinality dimension columns; same generator, same 52M elements.
Apple M4 Max/ 64 GB; median of 5 runs inside one `clickhouse local` process;
each version reads data it generated itself (content is deterministic and identical). Latest
published patch of each series:

| # | query shape | 25.8.28.1 | 25.9.7.56 | 25.10.7.6 | 25.12.11.4 | 26.1.12.23 | 26.3.17.56 | 26.6.2.81 |
|---|---|---|---|---|---|---|---|---|
| q1 | `ARRAY JOIN arr AS x WHERE x = c`, `Array(String)` | 79 | 81 | 68 | 66 | **114** | 118 | 154 |
| q2 | same, `Array(LowCardinality(String))` | 82 | 87 | 86 | 82 | **146** | 147 | 144 |
| q3 | `ARRAY JOIN measures WHERE measures.name = c` (Nested) | 206 | **261** | 257 | 250 | 260 | 261 | **387** |
| q4 | q3 with `max_threads = 1` | 759 | **930** | 907 | 916 | **1039** | 1065 | **1402** |
| q5 | `sum(arrayCount(x -> x = c, measures.name))` — no ARRAY JOIN | 254 | 253 | 253 | 247 | **311** | 327 | 324 |
| q6 | full dashboard query (time filter, `machine LIKE`, ORDER BY, 9 output columns) | 239 | 247 | 241 | 233 | **308** | 314 | 362 |

(ms; bold = where a step lands. 25.11 not tested: 25.12 is clean for the 26.1 step and the
25.9 step had already landed.)

### How to reproduce

## How to reproduce

Generate deterministic data (identical content on every version) and run one
query per version — official prebuilt binaries, macos-aarch64:

```sql
-- clickhouse local --path
CREATE TABLE t (arr Array(String)) ENGINE = MergeTree ORDER BY tuple();
INSERT INTO t

SELECT arrayMap(i -> concat('some.metric.name.of.realistic.len.', toString(cityHash64(number, i) % 15000)),
range(if(cityHash64(number, 5) % 10 = 0, 726, 16)))
FROM numbers(600000);
OPTIMIZE TABLE t FINAL;
-- 52,026,760 array elements total; ~15k distinct ~35-char strings; skewed lengths (90% × 16, 10% × 726).
-- The filter matches 3,420 elements (dashboards routinely filter on rare or absent names):
SELECT count() FROM t ARRAY JOIN arr AS x WHERE x = 'some.metric.name.of.realistic.len.7777';
```

This query (q1 below) runs in **66 ms on v25.12.11.4** and **114 ms on v26.1.12.23** (1.7×) on
the same machine.

[queries.sql](https://github.com/user-attachments/files/30273966/queries.sql)
[repro.sql](https://github.com/user-attachments/files/30273968/repro.sql)
[run.sh](https://github.com/user-attachments/files/30273967/run.sh)

Reading the steps:

- **25.9** hits only the Nested shapes (q3: 206→261, q4: 759→930); q1/q2/q5/q6 are flat.
- **26.1** hits every string-filter shape: q1 +73%, q2 +78%, q5 +26%, q6 +32% vs 25.12, and
stacks another +13% onto Nested q4. q5 shows it is not specific to ARRAY JOIN — the same
work as an `arrayCount` lambda regresses identically.
- **26.6** adds a Nested(+LowCardinality)-specific step: q3 261→387, q4 1065→1402.
- Aside: 25.10 *improved* q1 (79→68 ms), which the 26.1 step then more than undoes.

### Expected performance

Query latency on 26.x comparable to 25.8/25.12 for ARRAY JOIN / array-filter workloads.
The 26.1 step currently blocks our 26.3 LTS upgrade (dashboard p50 is dominated by exactly
this query class).

### Related issues and pull requests

The profile resembles #91430, but this repro contains no `multiIf` and the analyzer is on in both versions.

### Additional context

- `SETTINGS enable_analyzer = 0` on 26.3 recovers the Nested shape only partially
(261 → 233 ms; 25.8 is 206 ms). `compatibility = '24.3'` had no effect in our full
evaluation. Both versions run the new analyzer by default, so this is not a
settings-default change.
- In our production-shaped evaluation (45 real dashboard queries, restored production data,
24.3 baseline): all `ARRAY JOIN measures` queries were 1.2–1.6× slower on 26.3/26.6 with
identical result rows, while peak memory of heavy aggregations dropped ~4× (164 → 43 MB on
26.6).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.