ClickHouse / ClickHouse/ClickHouse
PARTITION BY key IF condition
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
_No response_
### Use case
When using hot/warm/cold architecture (https://clickhouse.com/docs/guides/developer/ttl#implementing-a-hotwarmcold-architecture) it makes sense to use a relatively high cardinality partitioning key
E.g. I want to store last 14 days on hot volume and then move data to cold volume. It makes sense to use toDate(timestamp) key
But after 14 days the partition will be moved and this partitioning key loses it's value
### Describe the solution you'd like
Add a condition to partition key
PARTITION BY toDate(timestamp) as key IF key > now() - INTERVAL 15 DAY
Clickhouse periodically checks the condition and when the condition becomes false clickhouse no longer respects this partitioning key (assumes it's value is null) and can merge parts with this key with other larts that have null key
This conversion is one way and shouldn't be reverted
Additionaly, this can be extended with multilevel partitioning strategies:
PARTITION BY toStartOfDay(timestamp) as day_key IF day_key > now() - INTERVAL 15 DAY
REPARTITION BY toStartOfMonth(day_key) as month_key IF month_key > now() - INTERVAL 366 DAY
Higher level strategy can only use previous level key to avoid complex repartitioning
### Describe alternatives you've considered
An alternative way to achieve something like that would be to create 2 identical tables, where one table is stored on cold_volume and is a materialized view of another table, which is stored on hot volume and is partitioned. The partitioned table on hot volume then can have a TTL DELETE strategy
Drawbacks: recent data is stored in 2 places, select queries become more complex. Schema changes and other modifications become more complex
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.