ClickHouse / ClickHouse/ClickHouse
Clickhouse batch inserts getting slowed(not running)
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
Capmint
### Describe the situation
Facing issue with system.distribution_queue having data files stuck for two days.
`WITH (
SELECT sum(bytes_on_disk) / sum(rows)
FROM system.parts
WHERE (database = 'db') AND (`table` = 'table_dist') AND (active = 1)
) AS avg_bpr
SELECT
data_path,
data_files,
data_compressed_bytes,
round(data_compressed_bytes / avg_bpr) AS est_rows_pending,
error_count,
last_exception_time
FROM system.distribution_queue
WHERE (database = 'db') AND (`table` = 'table_dist')
ORDER BY last_exception_time DESC
Query id: bb1e8f34-98bc-44f4-b0ad-30ba48bad837
┌─data_path───────────────────────────────────────────────────────────────────────────────┬─data_files─┬─data_compressed_bytes─┬─est_rows_pending─┬─error_count─┬─last_exception_time─┐
1. │ /var/lib/clickhouse/store/e16/e16b949d-8a01-4a7f-a4e4-61c7a2aa51c6/shard2_all_replicas/ │ 7080 │ 32344492 │ 4762301 │ 1297 │ 2025-09-03 16:29:51 │
2. │ /var/lib/clickhouse/store/e16/e16b949d-8a01-4a7f-a4e4-61c7a2aa51c6/shard3_all_replicas/ │ 7017 │ 33678636 │ 4958736 │ 1364 │ 2025-09-03 16:29:46 │
3. │ /var/lib/clickhouse/store/e16/e16b949d-8a01-4a7f-a4e4-61c7a2aa51c6/shard1_all_replicas/ │ 7033 │ 31549883 │ 4645305 │ 1303 │ 2025-09-03 16:29:25`
And
`MARKET_DATA_TS_sh3r3 :) SYSTEM FLUSH DISTRIBUTED db.table_dist;
SYSTEM FLUSH DISTRIBUTED db.table_dist
Query id: 86457586-f180-4674-9408-586c88918ffd
Elapsed: 121.836 sec.
Received exception from server (version 25.1.7):
Code: 159. DB::Exception: Received from localhost:9000. DB::Exception: Received from 10.40.210.148:9000. DB::Exception: Timeout exceeded: elapsed 60152.909755 ms, maximum: 60000 ms. (TIMEOUT_EXCEEDED)`
It is a setup of 2replicas and 3 shards. I am getting frequent timeouts at clickhouse keeper in server logs.
### Which ClickHouse versions are affected?
25.1.7.20
### How to reproduce
Create 3 shard 2 replica setup and add 5000 rows per sql query in a minute for 6 hours. Its been more than two days and queries are still stuck.
### Expected performance
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.