ClickHouse / ClickHouse/ClickHouse

Vertical merges doesn't work with AggregatingMergeTree

Open
#70,334 2 comments 0 reactions 0 assignees View on GitHub
comp-mergetree external performance
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

**Describe what's wrong**

Hi, it seems like Vertical merges doesn't work with AggregatingMergeTree

Clickhouse ignores settings and uses Horizontal alg, with real dataset it causes very high peak memory usage, and clickhouse merges fails with memory limit error.

I hope to reduce merge memory usage by switching merges to Vertical algorithm, but i have problems with it

**Does it reproduce on the most recent release?**

All releases

**How to reproduce**

```
select version()
┌─version()─┐
1. │ 24.6.2.17 │
└───────────┘
```
create table for data -
```
CREATE TABLE test_products_table(
date Date,
category_id UInt64,
product_id UInt64,
orders_count UInt32
)
ENGINE = ReplacingMergeTree
PARTITION BY toYYYYMM(date)
PRIMARY KEY (date, category_id, product_id)
ORDER BY (date, category_id, product_id)
```
create aggregating table with settings (to use vertical merge) -
```
CREATE TABLE test_products_aggregating_table(
date_from Date,
date_to Date,
category_id UInt64,
products_count AggregateFunction(uniq, UInt64),
products_with_orders_count AggregateFunction(uniqIf, UInt64, Bool)
)
ENGINE = AggregatingMergeTree
PRIMARY KEY (date_from, date_to, category_id)
ORDER BY (date_from, date_to, category_id)
PARTITION BY toYYYYMM(date_from)
SETTINGS
enable_vertical_merge_algorithm = 1,
vertical_merge_algorithm_min_rows_to_activate = 0,
vertical_merge_algorithm_min_bytes_to_activate = 0,
vertical_merge_algorithm_min_columns_to_activate = 0
```
create MV for filling aggregating table -
```
CREATE MATERIALIZED VIEW test_products_aggregating_table_by_week_mv
TO test_products_aggregating_table
AS
SELECT
toStartOfWeek(date, 3) date_from,
toLastDayOfWeek(date, 3) date_to,
category_id,
uniqState(product_id) as products_count,
uniqIfState(product_id, orders_count > 0) as products_with_orders_count
FROM test_products_table
GROUP BY date_from, date_to, category_id
```
fill product table -
```
INSERT INTO test_products_table
SELECT *
FROM VALUES(
'date Date, category_id UInt64, product_id UInt64, orders_count UInt32',
(toDate('2024-09-01'), 1, 1, 1),
(toDate('2024-09-01'), 1, 2, 0),
(toDate('2024-09-01'), 1, 3, 50),
(toDate('2024-09-01'), 2, 4, 0),
(toDate('2024-09-01'), 2, 5, 20),
(toDate('2024-09-01'), 2, 6, 0),
(toDate('2024-09-02'), 1, 1, 12),
(toDate('2024-09-02'), 1, 2, 0),
(toDate('2024-09-02'), 1, 3, 52),
(toDate('2024-09-02'), 2, 4, 0),
(toDate('2024-09-02'), 2, 5, 22),
(toDate('2024-09-02'), 2, 6, 0)
)

INSERT INTO test_products_table
SELECT *
FROM VALUES(
'date Date, category_id UInt64, product_id UInt64, orders_count UInt32',
(toDate('2024-09-03'), 1, 1, 1),
(toDate('2024-09-03'), 1, 2, 0),
(toDate('2024-09-03'), 1, 3, 50),
(toDate('2024-09-03'), 2, 4, 0),
(toDate('2024-09-03'), 2, 5, 20),
(toDate('2024-09-03'), 2, 6, 0),
(toDate('2024-09-04'), 1, 1, 12),
(toDate('2024-09-04'), 1, 2, 0),
(toDate('2024-09-04'), 1, 3, 52),
(toDate('2024-09-04'), 2, 4, 0),
(toDate('2024-09-04'), 2, 5, 22),
(toDate('2024-09-04'), 2, 6, 0)
)
```
check aggregating data inserted -
```
select
date_from,
date_to,
category_id,
uniqMerge(products_count) products_count,
uniqIfMerge(products_with_orders_count) products_with_orders_count
from
test_products_aggregating_table
group by date_from, date_to, category_id
order by date_from, date_to, category_id
```
```
┌──date_from─┬────date_to─┬─category_id─┬─products_count─┬─products_with_orders_count─┐
1. │ 2024-08-26 │ 2024-09-01 │ 1 │ 3 │ 2 │
2. │ 2024-08-26 │ 2024-09-01 │ 2 │ 3 │ 1 │
3. │ 2024-09-02 │ 2024-09-08 │ 1 │ 3 │ 2 │
4. │ 2024-09-02 │ 2024-09-08 │ 2 │ 3 │ 1 │
└────────────┴────────────┴─────────────┴────────────────┴────────────────────────────┘
```
check we have > 1 aggregating table active parts -
```
select
parts,
active_parts,
total_marks
from system.tables
where
name = 'test_products_aggregating_table'
order by engine, name
```
```
┌─parts─┬─active_parts─┬─total_marks─┐
1. │ 3 │ 3 │ 6 │
└───────┴──────────────┴─────────────┘
```
force merges on aggregating table -
```
optimize table test_products_aggregating_table final
```
check we have lower active parts -
```
┌─parts─┬─active_parts─┬─total_marks─┐
1. │ 5 │ 2 │ 10 │
└───────┴──────────────┴─────────────┘
```

check if Vertical merge alg applied (it is not) -
```
select
event_type,
merge_reason,
merge_algorithm merge_alg,
event_time,
round(duration_ms / 1000, 2) dur_sec,
part_type,
formatReadableSize(peak_memory_usage) pmu,
length(merged_from) parts_count,
error
from system.part_log
where
event_type = 'MergeParts'
and table = 'test_products_aggregating_table'
order by event_time desc
limit 10
```
```
┌─event_type─┬─merge_reason─┬─merge_alg──┬──────────event_time─┬─dur_sec─┬─part_type─┬─pmu──────┬─parts_count─┬─error─┐
1. │ MergeParts │ RegularMerge │ Horizontal │ 2024-10-03 16:20:17 │ 0.01 │ Compact │ 3.24 MiB │ 2 │ 0 │
```

so as we see it uses Horizontal instead of Vertical merge merge_algorithm

**Additional context**
I tried to resolve my high memory usage problem in another way - `merge_max_block_size_bytes = 10485`. it helps, but only up to a certain amount of data.
In real dataset we have more aggregates (at least 12 instead of 2 in issue example) and huge table with data

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.