ClickHouse / ClickHouse/ClickHouse
Timeout during lightweight delete
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
**Company or project name**
SME high data throughput application.
**Describe the unexpected behaviour**
We periodically need to delete data from a ReplacingMergeTree table with 83 billion rows at the moment.
In this particular example the delete would result in 91 million deleted rows, however a lightweight delete simply times out (a heavyweight does as well).
This is the corresponding entry in system.mutations:
database: default
table: time_series_data
mutation_id: mutation_128759.txt
command: UPDATE _row_exists = 0 WHERE (first_id, second_id) IN (SELECT first_id, second_id FROM default.metadata WHERE common_id IN ['123456'])
create_time: 2024-11-11 17:05:49
block_numbers.partition_id: ['']
block_numbers.number: [128759]
parts_to_do_names: ['all_108114_108709_4']
parts_to_do: 1
is_done: 0
latest_failed_part: all_108114_108709_4
latest_fail_time: 2024-11-12 08:27:44
latest_fail_reason: Code: 159. DB::Exception: Timeout exceeded: elapsed 2354.879888484 seconds, maximum: 600: While executing MergeTreeInOrder. (TIMEOUT_EXCEEDED) (version 23.8.16.16 (official build))
It started off with 16 parts, finishing 15 after a few minutes and then failed to delete the last one over night. We append `SETTINGS max_execution_time = 0` to the query, but that has seemingly no effect. The rest of the query is simply a `DELETE FROM time_series_data WHERE (first_id, second_id) IN (SELECT first_id, second_id FROM default.metadata WHERE common_id IN ['123456']`. Partitioning the data on common_id would be quite a large operation we would prefer to avoid.
Counting rows matching that WHERE clause after 12 hours still yields the pre-delete count.
**How to reproduce**
* Clickhouse Version: 23.8
* HTTP interface
* We have global max_execution_time setting set to 600s
**Expected behavior**
Why can a delete/merge operation even time out? I expect the query's max_execution_time setting to be ignored, but why isn't the global setting ignored?
Contributor guide
Assessment
This issue has not been assessed yet.