ClickHouse / ClickHouse/ClickHouse

clickhouse write latency gradually increased.

Open
#40,784 0 comments 0 reactions 0 assignees View on GitHub
comp-mergetree external performance
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

(you don't have to strictly follow this form)

**Describe the situation**
I have 4 machine, which write data to clickhouse every 10w row, each machine have write upperlimit 80w.
![截屏2022-08-30 下午3 23 28](https://user-images.githubusercontent.com/52686065/187375377-6fb0a293-5c2b-4cfa-a952-83bafc9391cf.png)

i found that the clickhouse p99 write latency will gradually increased.
you can see that in 14.50, the write latency is 10s, but it gradually increased to 50 second.
In 15:20, i restart the machine, the write latency immediate back to 10s.
![截屏2022-08-30 下午3 27 01](https://user-images.githubusercontent.com/52686065/187376054-c894c566-414e-496b-8940-38fdc6af8c7c.png)

**How to reproduce**
* clickhouse v21.8
* [[click](https://github.com/ClickHouse/clickhouse-go) v](https://github.com/ClickHouse/clickhouse-go/releases/tag/v2.3.0)

**Expected performance**
the write latency should not increased over time passed by.

**Additional context**
My clickhouse server have 3 node, each of them is 32core 64gb 20tb ssd.

here is the cpu and memory usage of clickhouse.
![截屏2022-08-30 下午3 30 57](https://user-images.githubusercontent.com/52686065/187376854-e8c953bc-cc82-4a75-971b-8b719b9ee8d7.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.