ClickHouse / ClickHouse/ClickHouse

[RFC] Inroduce a new kind of background task between merge and mutation (rework TTLs)

Open
#74,694 5 comments 8 reactions 1 assignee Claimed by @CurtizJ View on GitHub
comp-ttl feature performance
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

ClickHouse Inc.

### Use case

More efficient TTLs. Automatic materialization of lightweight deletes and updates in parts that don't participate in merges.

### Describe the solution you'd like

Introduce a new kind of task that can rewrite a single part and apply changes to it without increasing its mutation version. Regarding the ClickHouse codebase, it will be a `MutateTask` which increases the level of the part instead of the mutation version.

Introduce a background thread that schedules such tasks for parts according to certain conditions (e.g., expired TTL). Settings similar to `merge_with_ttl_timeout` and `max_number_of_merges_with_ttl_in_pool` will limit the number of these tasks. Also, according to Merge Selector's heuristics, tasks shouldn't be scheduled for parts likely to be merged soon.

Rework the existing TTL tasks to this interface:
- `TTL DELETE` with `ttl_only_drop_parts`: the task will replace part to empty part.
- `TTL DELETE`, `TTL GROUP BY`: the task will apply TTL algorithm and rewrite part.
- `COLUMN TTL` with `ttl_only_drop_parts`: the task will run `CLEAR COLUMN` for the part.
- `COLUMN TTL`: the task will update and rewrite the required column in part.

Introduce new tasks:
- Materialization of lightweight deletes: the task will apply the `_row_exists` mask if the number of deleted rows exceeds the threshold specified in table settings. It's also possible to set some TTL for this, which will be counted since the `_row_exists` is written.
- Materialization of lightweight updates: the same as above but for lightweight updates. The task will apply patch parts.

All of these tasks could still be applied to parts during regular merges scheduled by a common Merge Selector.

### Describe alternatives you've considered
For column TTL, manually run `ALTER TABLE ... CLEAR COLUMN ... IN PARTITION`.
For materialization of lightweight deletes manually run `ALTER TABLE ... APPLY DELETED MASK`.
For materialization of lightweight updates manually run `ALTER TABLE ... APPLY PATCHES`.

### Additional context

It will make the column's TTLs more efficient and usable on a high workload. Also it will make TTL with `ttl_only_drop_parts` more efficient because it will be applied faster (now it respects setting `merge_with_ttl_timeout` and `max_number_of_merges_with_ttl_in_pool` which is not actually needed).

Refs: #26331, #26328, #71196, #56728, #45397.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.