ClickHouse / ClickHouse/ClickHouse

Clickhouse batch inserts getting slowed(not running)

Open
#86,634 1 comment 2 reactions 0 assignees View on GitHub
comp-distributed external performance
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

Capmint

### Describe the situation

Facing issue with system.distribution_queue having data files stuck for two days.

`WITH (
SELECT sum(bytes_on_disk) / sum(rows)
FROM system.parts
WHERE (database = 'db') AND (`table` = 'table_dist') AND (active = 1)
) AS avg_bpr
SELECT
data_path,
data_files,
data_compressed_bytes,
round(data_compressed_bytes / avg_bpr) AS est_rows_pending,
error_count,
last_exception_time
FROM system.distribution_queue
WHERE (database = 'db') AND (`table` = 'table_dist')
ORDER BY last_exception_time DESC

Query id: bb1e8f34-98bc-44f4-b0ad-30ba48bad837

┌─data_path───────────────────────────────────────────────────────────────────────────────┬─data_files─┬─data_compressed_bytes─┬─est_rows_pending─┬─error_count─┬─last_exception_time─┐
1. │ /var/lib/clickhouse/store/e16/e16b949d-8a01-4a7f-a4e4-61c7a2aa51c6/shard2_all_replicas/ │ 7080 │ 32344492 │ 4762301 │ 1297 │ 2025-09-03 16:29:51 │
2. │ /var/lib/clickhouse/store/e16/e16b949d-8a01-4a7f-a4e4-61c7a2aa51c6/shard3_all_replicas/ │ 7017 │ 33678636 │ 4958736 │ 1364 │ 2025-09-03 16:29:46 │
3. │ /var/lib/clickhouse/store/e16/e16b949d-8a01-4a7f-a4e4-61c7a2aa51c6/shard1_all_replicas/ │ 7033 │ 31549883 │ 4645305 │ 1303 │ 2025-09-03 16:29:25`

And

`MARKET_DATA_TS_sh3r3 :) SYSTEM FLUSH DISTRIBUTED db.table_dist;

SYSTEM FLUSH DISTRIBUTED db.table_dist

Query id: 86457586-f180-4674-9408-586c88918ffd

Elapsed: 121.836 sec.

Received exception from server (version 25.1.7):
Code: 159. DB::Exception: Received from localhost:9000. DB::Exception: Received from 10.40.210.148:9000. DB::Exception: Timeout exceeded: elapsed 60152.909755 ms, maximum: 60000 ms. (TIMEOUT_EXCEEDED)`

It is a setup of 2replicas and 3 shards. I am getting frequent timeouts at clickhouse keeper in server logs.

### Which ClickHouse versions are affected?

25.1.7.20

### How to reproduce

Create 3 shard 2 replica setup and add 5000 rows per sql query in a minute for 6 hours. Its been more than two days and queries are still stuck.

### Expected performance

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.