ClickHouse / ClickHouse/ClickHouse

Some mutations can’t be cancelled during execution, causing potential replication issues

Open
#78,497 4 comments 0 reactions 0 assignees View on GitHub
comp-mutations external unexpected behaviour
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

_No response_

### Describe the unexpected behaviour

When a slow mutation (such as an ALTER DELETE that removes almost all rows from a large data part) is in progress, it sometimes cannot be cancelled even if a cancellation request is issued.

This inability to stop a mutation becomes problematic during replication. For example, if a ZooKeeper hardware exception triggers an automatic replica restart, the restart process waits for all mutations to be killed, leaving the table in a read-only state until the ongoing mutation completes.

### How to reproduce

See the repro in gist: https://gist.github.com/filimonov/b73f7547ea6652149b8551be43c46d3b

### Expected behavior

_No response_

### Error message and/or stacktrace

_No response_

### Additional context

A likely cause is that the cancellation check in [MutateTask.cpp](https://github.com/ClickHouse/ClickHouse/blob/18df3d6f1bf19fb70a3fee5cd22a39ab981b279a/src/Storages/MergeTree/MutateTask.cpp) is only performed at the boundaries of output blocks. When the mutation results in an empty block (because all rows are removed), the cancellation check never happens.

An easy way to confirm this is to adjust the mutation so that it produces some rows; in that case, the mutation cancellation works properly

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.