ClickHouse / ClickHouse/ClickHouse

Alter table attach / move / replace partition should clean TTL info from the attached parts

Open
#50,610 1 comment 2 reactions 0 assignees View on GitHub
comp-ttl external unexpected behaviour
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

https://fiddle.clickhouse.com/3dd9704d-2a0f-475a-96ff-49c706cd7390

data is removed from the table with `TTL ts + interval 100 day` because the attached part has another TTL info `TTL ts + interval 1 hour`

```sql
CREATE TABLE users (uid Int64, ts DateTime)
ENGINE=MergeTree partition by toYYYYMMDD(ts) order by uid
TTL ts + interval 1 hour;

CREATE TABLE users_history (uid Int64, ts DateTime)
ENGINE=MergeTree partition by toYYYYMMDD(ts) order by uid
TTL ts + interval 100 day;

insert into users select number, now()- interval 59 minute - interval 58 second from numbers(10000);

alter table users move partition tuple(toYYYYMMDD(now())) to table users_history;

select count() from users_history;
┌─count()─┐
│ 10000 │
└─────────┘

select sleep(3) format Null;

optimize table users_history final;

select count() from users_history;
┌─count()─┐
│ 0 │ -- data is removed by TTL
└─────────┘

select delete_ttl_info_min, delete_ttl_info_max, rows, name
from system.parts
where table = 'users_history' and active ;
┌─delete_ttl_info_min─┬─delete_ttl_info_max─┬─rows─┬─name───────────┐
│ 1970-01-01 00:00:00 │ 1970-01-01 00:00:00 │ 0 │ 20230605_1_1_1 │
└─────────────────────┴─────────────────────┴──────┴────────────────┘
```

Or TTL remover should check the current TTL rule on the part somehow? Validating a single row?

The same behavior with a single table (materialize_ttl_after_modify=0) https://fiddle.clickhouse.com/00f5d9e3-3548-4e3e-a40c-096495c67b15

```sql
CREATE TABLE users (uid Int64, ts DateTime)
ENGINE=MergeTree partition by toYYYYMMDD(ts) order by uid
TTL ts + interval 1 hour;

insert into users select number, now()- interval 59 minute - interval 58 second from numbers(10000);

select count() from users;
┌─count()─┐
│ 10000 │
└─────────┘

set materialize_ttl_after_modify =0;

alter table users modify TTL ts + INTERVAL 100 day;

select sleep(3) format Null;

optimize table users final;

select count() from users;
┌─count()─┐
│ 0 │
└─────────┘
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.