ClickHouse / ClickHouse/ClickHouse

Changing RECOMPRESS TTL on existing table does not recompress needed parts.

Open
#64,841 0 comments 0 reactions 0 assignees View on GitHub
comp-ttl external unexpected behaviour
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

**Describe the unexpected behaviour**
Changing RECOMPRESS TTL on existing table does not recompress needed parts. Some parts recompress back to LZ4 and they shouldn't.

**How to reproduce**
* Which ClickHouse server version to use
24.5.1.1763

https://fiddle.clickhouse.com/501289a6-3caa-4012-9681-d0a3ffb92ba8

```
CREATE TABLE recompress_test
(
a UInt64,
b String,
c String,
d String,
e String,
f String,
date Date
) Engine = MergeTree
PARTITION BY toYYYYMM(date)
ORDER BY a;

INSERT INTO recompress_test
SELECT number, rand()::String, rand()::String, rand()::String, rand()::String, rand()::String,
now() - INTERVAL rand() % 365 DAY
FROM numbers(10e5);

ALTER TABLE recompress_test
modify TTL
date + INTERVAL 5 MONTH RECOMPRESS CODEC(ZSTD(1));

select partition, name, default_compression_codec, part_type,
recompression_ttl_info.expression from system.parts
where table='recompress_test'
and active
ORDER BY name;

/* this is fine
202306 202306_12_12_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202307 202307_4_4_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202308 202308_6_6_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202309 202309_11_11_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202310 202310_10_10_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202311 202311_8_8_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202312 202312_1_1_1_14 ZSTD(1) Compact ['date + toIntervalMonth(5)']
202401 202401_9_9_0_14 LZ4 Compact ['date + toIntervalMonth(5)']
202402 202402_5_5_0_14 LZ4 Compact ['date + toIntervalMonth(5)']
202403 202403_3_3_0_14 LZ4 Compact ['date + toIntervalMonth(5)']
202404 202404_7_7_0_14 LZ4 Compact ['date + toIntervalMonth(5)']
202405 202405_2_2_0_14 LZ4 Compact ['date + toIntervalMonth(5)']
202406 202406_13_13_0_14 LZ4 Compact ['date + toIntervalMonth(5)']
*/

ALTER TABLE recompress_test
modify TTL
date + INTERVAL 2 MONTH RECOMPRESS CODEC(ZSTD(2));

select partition, name, default_compression_codec, part_type,
recompression_ttl_info.expression from system.parts
where table='recompress_test'
and active
ORDER BY name;

/* after modify TTL only three parts got recompressed to ZSTD(2)
older parts got recompressed back to LZ4
202306 202306_12_12_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202307 202307_4_4_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202308 202308_6_6_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202309 202309_11_11_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202310 202310_10_10_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202311 202311_8_8_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202312 202312_1_1_1_15 LZ4 Compact ['date + toIntervalMonth(2)']
202401 202401_9_9_1_15 ZSTD(2) Compact ['date + toIntervalMonth(2)']
202402 202402_5_5_1_15 ZSTD(2) Compact ['date + toIntervalMonth(2)']
202403 202403_3_3_1_15 ZSTD(2) Compact ['date + toIntervalMonth(2)']
202404 202404_7_7_0_15 LZ4 Compact ['date + toIntervalMonth(2)']
202405 202405_2_2_0_15 LZ4 Compact ['date + toIntervalMonth(2)']
202406 202406_13_13_0_15 LZ4 Compact ['date + toIntervalMonth(2)']
*/
```

**Expected behavior**
With default settings `materialize_ttl_after_modify=1` and `materialize_ttl_recalculate_only=0` parts should be recompressed when executing `ALTER TABLE MODIFY TTL`.

**Additional info**
Running `ALTER TABLE MATERIALIZE TTL` fixes the issue.

If those settings are applied:

```
SET materialize_ttl_after_modify=0;

ALTER TABLE recompress_test MODIFY SETTING materialize_ttl_recalculate_only=1;
```

Then first run of `MATERIALIZE TTL` reproduces the same behaviour, second run of `MATERIALIZE TTL` fixes it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.