ClickHouse / ClickHouse/ClickHouse

PARTITION BY key IF condition

Open
#102,617 1 comment 1 reaction 0 assignees View on GitHub
comp-mergetree external feature
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

_No response_

### Use case

When using hot/warm/cold architecture (https://clickhouse.com/docs/guides/developer/ttl#implementing-a-hotwarmcold-architecture) it makes sense to use a relatively high cardinality partitioning key
E.g. I want to store last 14 days on hot volume and then move data to cold volume. It makes sense to use toDate(timestamp) key
But after 14 days the partition will be moved and this partitioning key loses it's value

### Describe the solution you'd like

Add a condition to partition key
PARTITION BY toDate(timestamp) as key IF key > now() - INTERVAL 15 DAY

Clickhouse periodically checks the condition and when the condition becomes false clickhouse no longer respects this partitioning key (assumes it's value is null) and can merge parts with this key with other larts that have null key

This conversion is one way and shouldn't be reverted

Additionaly, this can be extended with multilevel partitioning strategies:
PARTITION BY toStartOfDay(timestamp) as day_key IF day_key > now() - INTERVAL 15 DAY
REPARTITION BY toStartOfMonth(day_key) as month_key IF month_key > now() - INTERVAL 366 DAY

Higher level strategy can only use previous level key to avoid complex repartitioning

### Describe alternatives you've considered

An alternative way to achieve something like that would be to create 2 identical tables, where one table is stored on cold_volume and is a materialized view of another table, which is stored on hot volume and is partitioned. The partitioned table on hot volume then can have a TTL DELETE strategy

Drawbacks: recent data is stored in 2 places, select queries become more complex. Schema changes and other modifications become more complex

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.