ClickHouse / ClickHouse/ClickHouse
Changing RECOMPRESS TTL on existing table does not recompress needed parts.
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
**Describe the unexpected behaviour**
Changing RECOMPRESS TTL on existing table does not recompress needed parts. Some parts recompress back to LZ4 and they shouldn't.
**How to reproduce**
* Which ClickHouse server version to use
24.5.1.1763
https://fiddle.clickhouse.com/501289a6-3caa-4012-9681-d0a3ffb92ba8
```
CREATE TABLE recompress_test
(
a UInt64,
b String,
c String,
d String,
e String,
f String,
date Date
) Engine = MergeTree
PARTITION BY toYYYYMM(date)
ORDER BY a;
INSERT INTO recompress_test
SELECT number, rand()::String, rand()::String, rand()::String, rand()::String, rand()::String,
now() - INTERVAL rand() % 365 DAY
FROM numbers(10e5);
ALTER TABLE recompress_test
modify TTL
date + INTERVAL 5 MONTH RECOMPRESS CODEC(ZSTD(1));
select partition, name, default_compression_codec, part_type,
recompression_ttl_info.expression from system.parts
where table='recompress_test'
and active
ORDER BY name;
/* this is fine
202306 202306_12_12_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202307 202307_4_4_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202308 202308_6_6_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202309 202309_11_11_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202310 202310_10_10_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202311 202311_8_8_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202312 202312_1_1_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202401 202401_9_9_0_14 LZ4 Compact ['date + toIntervalMonth(5)']
202402 202402_5_5_0_14 LZ4 Compact ['date + toIntervalMonth(5)']
202403 202403_3_3_0_14 LZ4 Compact ['date + toIntervalMonth(5)']
202404 202404_7_7_0_14 LZ4 Compact ['date + toIntervalMonth(5)']
202405 202405_2_2_0_14 LZ4 Compact ['date + toIntervalMonth(5)']
202406 202406_13_13_0_14 LZ4 Compact ['date + toIntervalMonth(5)']
*/
ALTER TABLE recompress_test
modify TTL
date + INTERVAL 2 MONTH RECOMPRESS CODEC(ZSTD(2));
select partition, name, default_compression_codec, part_type,
recompression_ttl_info.expression from system.parts
where table='recompress_test'
and active
ORDER BY name;
/* after modify TTL only three parts got recompressed to ZSTD(2)
older parts got recompressed back to LZ4
202306 202306_12_12_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202307 202307_4_4_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202308 202308_6_6_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202309 202309_11_11_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202310 202310_10_10_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202311 202311_8_8_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202312 202312_1_1_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202401 202401_9_9_1_15 ZSTD(2) Compact ['date + toIntervalMonth(2)']
202402 202402_5_5_1_15 ZSTD(2) Compact ['date + toIntervalMonth(2)']
202403 202403_3_3_1_15 ZSTD(2) Compact ['date + toIntervalMonth(2)']
202404 202404_7_7_0_15 LZ4 Compact ['date + toIntervalMonth(2)']
202405 202405_2_2_0_15 LZ4 Compact ['date + toIntervalMonth(2)']
202406 202406_13_13_0_15 LZ4 Compact ['date + toIntervalMonth(2)']
*/
```
**Expected behavior**
With default settings `materialize_ttl_after_modify=1` and `materialize_ttl_recalculate_only=0` parts should be recompressed when executing `ALTER TABLE MODIFY TTL`.
**Additional info**
Running `ALTER TABLE MATERIALIZE TTL` fixes the issue.
If those settings are applied:
```
SET materialize_ttl_after_modify=0;
ALTER TABLE recompress_test MODIFY SETTING materialize_ttl_recalculate_only=1;
```
Then first run of `MATERIALIZE TTL` reproduces the same behaviour, second run of `MATERIALIZE TTL` fixes it.
Contributor guide
Assessment
This issue has not been assessed yet.