ClickHouse / ClickHouse/ClickHouse

In-line FINAL vs. final=1 yield different SelectedBytes for Distributed ReplicatedAggregatingMergeTree Tables

Open
#110,101 1 comment 0 reactions 0 assignees View on GitHub
comp-distributed external performance
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

[Shopify](https://www.shopify.com/)

### Describe the situation

During performance testing, I noticed a discrepancy between using the `FINAL` keyword vs. `SETTINGS final=1`. A `Distributed` table was created on top of a `ReplicatedAggregatingMergeTree` table. The settings path completed ~60% faster by wall clock time (16s vs 28s) and less bytes scanned (95GB vs. 105GB). This query was routed to a single shard.

Caching may explain query time, but I don't believe that it should impact `SelectedBytes`. When I ran the shard-local query **with both methods**, bytes scanned were in-line with the `FINAL` keyword method with the `Distributed` table. Hashed results indicated that the output is identical.

`SETTINGS final=1` seems to have some odd interaction with `Distributed` tables over a sharded table using the `ReplicatedAggregatingMergeTree` engine.

### Which ClickHouse versions are affected?

26.3.10.62

### How to reproduce

```
-- Replace and with values for your environment.
CREATE TABLE .t_local ON CLUSTER ''
(
`a` UInt64,
`b` String,
`c` String,
`d` String,
`e` String,

`s1` SimpleAggregateFunction(min, Tuple(Bool, DateTime64(3))),
`s2` SimpleAggregateFunction(max, Tuple(Bool, Bool)),
`s3` SimpleAggregateFunction(max, Tuple(Bool, DateTime64(3), Float64)),
`s4` SimpleAggregateFunction(max, Tuple(Bool, DateTime64(3))),

`u1` SimpleAggregateFunction(max, Tuple(Bool, DateTime64(3), Float64)),
`u2` SimpleAggregateFunction(max, Tuple(Bool, DateTime64(3), Float64)),
`u3` SimpleAggregateFunction(max, Tuple(Bool, DateTime64(3), Float64)),
`u4` SimpleAggregateFunction(max, Tuple(Bool, DateTime64(3), Float64)),
`u5` SimpleAggregateFunction(max, Tuple(Bool, DateTime64(3), Float64)),
`u6` SimpleAggregateFunction(max, Tuple(Bool, DateTime64(3), Float64)),
`u7` SimpleAggregateFunction(max, Tuple(Bool, DateTime64(3), Float64)),
`u8` SimpleAggregateFunction(max, Tuple(Bool, DateTime64(3), Float64))
)
ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/{database}/{shard}/t_local', '{replica}')
ORDER BY (a, b, c, d, e)
PRIMARY KEY (a, b, c, d, e)
SETTINGS index_granularity = 8192;

CREATE TABLE .t_dist ON CLUSTER '' AS .t_local
ENGINE = Distributed('', '', 't_local', intHash64(a));
```

I used a script to insert in batches of 1m:
```
INSERT INTO .t_local
WITH
toUInt64() AS batch,
toUInt64(1000000) AS rows_per_batch,
toUInt64(250000) AS unique_keys,
toUInt64(number % unique_keys) AS k,
toUInt64(number + batch * rows_per_batch) AS v,
toDateTime64('2026-01-01 00:00:00', 3) + toIntervalSecond(v % (86400 * 180)) AS ts,
CAST(v % 10 != 0, 'Bool') AS p,
toFloat64(v % 10000) / 100.0 AS x
SELECT
toUInt64(42) AS a,
concat('b_', toString(k)) AS b,
concat('c_', toString(k % 1000)) AS c,
if(k % 2 = 0, 'd0', 'd1') AS d,
if(k % 3 = 0, 'e0', 'e1') AS e,

tuple(CAST(v % 10 = 0, 'Bool'), ts) AS s1,
tuple(p, CAST(v % 7 = 0, 'Bool')) AS s2,
tuple(p, ts, x) AS s3,
tuple(p, ts) AS s4,

tuple(p, ts, x + 1) AS u1,
tuple(p, ts, x + 2) AS u2,
tuple(p, ts, x + 3) AS u3,
tuple(p, ts, x + 4) AS u4,
tuple(p, ts, x + 5) AS u5,
tuple(p, ts, x + 6) AS u6,
tuple(p, ts, x + 7) AS u7,
tuple(p, ts, x + 8) AS u8
FROM numbers(rows_per_batch)
SETTINGS
max_threads = 16,
optimize_on_insert = 0;
```
for batches 0-3. You can verify the counts via:
```
SELECT
count(),
uniqExact(b)
FROM .t_dist;
```
which should yield:
```
4,000,000 inserted rows
250,000 unique logical keys
4 active parts, if each batch lands as one part
```
Then the A/B query structure is as follows. Comments show where you should modify the query for each side of the test. If you haven't stopped merges on your cluster, you'll want to run this query shortly after you populate the table:
```
/*
Variant A — query setting finalization:
- leave `FROM .t_dist`
- leave `SETTINGS final = 1`

Variant B — inline FINAL:
- change to `FROM .t_dist FINAL`
- remove the `SETTINGS final = 1` line
*/

WITH
toDateTime64('2026-01-01 00:00:00', 3) AS start_ts,
toDateTime64('2026-06-30 23:59:59', 3) AS end_ts,
q AS
(
SELECT
b,
tupleElement(s2, 2) AS m,
NOT tupleElement(s4, 1) AS f,
tupleElement(s1, 2) AS ts
FROM .t_dist /* add FINAL here for inline-FINAL variant */
WHERE a = 42
AND tupleElement(s1, 2) BETWEEN start_ts AND end_ts
)
SELECT
toNullable(toDate(ts)) AS day,
ifNull(uniqExactIf(b, m = true), 0) AS n,
1 AS calculated_row
FROM q
WHERE f = false
GROUP BY ALL
ORDER BY day WITH FILL
FROM toDate(start_ts)
TO date_add(DAY, 1, toDate(end_ts))
STEP INTERVAL 1 DAY
SETTINGS final = 1 -- remove this line for inline-FINAL variant
FORMAT Null;
```

### Expected performance

I would expect in-line `FINAL` and `final=1` query settings to have identical bytes scanned. In this synthetic test, we got the following:

```
A variant: Distributed table + SETTINGS final=1
A avg query_duration_ms: 361 ms
A avg SelectedBytes: 194.24 MiB
A avg read_rows: 4,229,376
A avg UserTimeMicroseconds: 20,000 µs

B variant: Distributed table + inline FINAL
B avg query_duration_ms: 496 ms
B avg SelectedBytes: 284.97 MiB
B avg read_rows: 4,229,376
B avg UserTimeMicroseconds: 34,000 µs

delta query_duration_ms: +135 ms inline FINAL
delta SelectedBytes: +90.73 MiB inline FINAL
delta read_rows: 0
delta UserTimeMicroseconds: +14,000 µs inline FINAL
```

### Related issues and pull requests

Related:

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.