ClickHouse / ClickHouse/ClickHouse
[RFC] Inroduce a new kind of background task between merge and mutation (rework TTLs)
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
ClickHouse Inc.
### Use case
More efficient TTLs. Automatic materialization of lightweight deletes and updates in parts that don't participate in merges.
### Describe the solution you'd like
Introduce a new kind of task that can rewrite a single part and apply changes to it without increasing its mutation version. Regarding the ClickHouse codebase, it will be a `MutateTask` which increases the level of the part instead of the mutation version.
Introduce a background thread that schedules such tasks for parts according to certain conditions (e.g., expired TTL). Settings similar to `merge_with_ttl_timeout` and `max_number_of_merges_with_ttl_in_pool` will limit the number of these tasks. Also, according to Merge Selector's heuristics, tasks shouldn't be scheduled for parts likely to be merged soon.
Rework the existing TTL tasks to this interface:
- `TTL DELETE` with `ttl_only_drop_parts`: the task will replace part to empty part.
- `TTL DELETE`, `TTL GROUP BY`: the task will apply TTL algorithm and rewrite part.
- `COLUMN TTL` with `ttl_only_drop_parts`: the task will run `CLEAR COLUMN` for the part.
- `COLUMN TTL`: the task will update and rewrite the required column in part.
Introduce new tasks:
- Materialization of lightweight deletes: the task will apply the `_row_exists` mask if the number of deleted rows exceeds the threshold specified in table settings. It's also possible to set some TTL for this, which will be counted since the `_row_exists` is written.
- Materialization of lightweight updates: the same as above but for lightweight updates. The task will apply patch parts.
All of these tasks could still be applied to parts during regular merges scheduled by a common Merge Selector.
### Describe alternatives you've considered
For column TTL, manually run `ALTER TABLE ... CLEAR COLUMN ... IN PARTITION`.
For materialization of lightweight deletes manually run `ALTER TABLE ... APPLY DELETED MASK`.
For materialization of lightweight updates manually run `ALTER TABLE ... APPLY PATCHES`.
### Additional context
It will make the column's TTLs more efficient and usable on a high workload. Also it will make TTL with `ttl_only_drop_parts` more efficient because it will be applied faster (now it respects setting `merge_with_ttl_timeout` and `max_number_of_merges_with_ttl_in_pool` which is not actually needed).
Refs: #26331, #26328, #71196, #56728, #45397.
Contributor guide
Assessment
This issue has not been assessed yet.