ClickHouse / ClickHouse/ClickHouse
Some mutations can’t be cancelled during execution, causing potential replication issues
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
_No response_
### Describe the unexpected behaviour
When a slow mutation (such as an ALTER DELETE that removes almost all rows from a large data part) is in progress, it sometimes cannot be cancelled even if a cancellation request is issued.
This inability to stop a mutation becomes problematic during replication. For example, if a ZooKeeper hardware exception triggers an automatic replica restart, the restart process waits for all mutations to be killed, leaving the table in a read-only state until the ongoing mutation completes.
### How to reproduce
See the repro in gist: https://gist.github.com/filimonov/b73f7547ea6652149b8551be43c46d3b
### Expected behavior
_No response_
### Error message and/or stacktrace
_No response_
### Additional context
A likely cause is that the cancellation check in [MutateTask.cpp](https://github.com/ClickHouse/ClickHouse/blob/18df3d6f1bf19fb70a3fee5cd22a39ab981b279a/src/Storages/MergeTree/MutateTask.cpp) is only performed at the boundaries of output blocks. When the mutation results in an empty block (because all rows are removed), the cancellation check never happens.
An easy way to confirm this is to adjust the mutation so that it produces some rows; in that case, the mutation cancellation works properly
Contributor guide
Assessment
This issue has not been assessed yet.