ClickHouse / ClickHouse/ClickHouse

Timeout during lightweight delete

Open
#71,796 1 comment 0 reactions 0 assignees View on GitHub
comp-mutations external unexpected behaviour
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

**Company or project name**

SME high data throughput application.

**Describe the unexpected behaviour**
We periodically need to delete data from a ReplacingMergeTree table with 83 billion rows at the moment.
In this particular example the delete would result in 91 million deleted rows, however a lightweight delete simply times out (a heavyweight does as well).

This is the corresponding entry in system.mutations:

database: default
table: time_series_data
mutation_id: mutation_128759.txt
command: UPDATE _row_exists = 0 WHERE (first_id, second_id) IN (SELECT first_id, second_id FROM default.metadata WHERE common_id IN ['123456'])
create_time: 2024-11-11 17:05:49
block_numbers.partition_id: ['']
block_numbers.number: [128759]
parts_to_do_names: ['all_108114_108709_4']
parts_to_do: 1
is_done: 0
latest_failed_part: all_108114_108709_4
latest_fail_time: 2024-11-12 08:27:44
latest_fail_reason: Code: 159. DB::Exception: Timeout exceeded: elapsed 2354.879888484 seconds, maximum: 600: While executing MergeTreeInOrder. (TIMEOUT_EXCEEDED) (version 23.8.16.16 (official build))

It started off with 16 parts, finishing 15 after a few minutes and then failed to delete the last one over night. We append `SETTINGS max_execution_time = 0` to the query, but that has seemingly no effect. The rest of the query is simply a `DELETE FROM time_series_data WHERE (first_id, second_id) IN (SELECT first_id, second_id FROM default.metadata WHERE common_id IN ['123456']`. Partitioning the data on common_id would be quite a large operation we would prefer to avoid.

Counting rows matching that WHERE clause after 12 hours still yields the pre-delete count.

**How to reproduce**
* Clickhouse Version: 23.8
* HTTP interface
* We have global max_execution_time setting set to 600s

**Expected behavior**
Why can a delete/merge operation even time out? I expect the query's max_execution_time setting to be ignored, but why isn't the global setting ignored?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.