Altinity / Altinity/clickhouse-operator
High query load from CHOP
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
Sometimes when CHOP is reconciling, it would send very high query load to each pod in the cluster.
e.g.: Below shows the top 5 queries received in 2 minutes by one ClickHouse node in a cluster of 120, all come from the CHOP
SELECT
query,
count() AS cnt
FROM system.query_log
WHERE (event_time >= toDateTime('2025-04-16 23:46:00')) AND (event_time <= toDateTime('2025-04-16 23:48:00'))
GROUP BY query
ORDER BY cnt DESC
LIMIT 5
Query id: 94e7460f-36db-4f99-9837-dc3040a2b9f6
┌─query────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───cnt─┐
1. │ SELECT DISTINCT `database`, `name`, `create_table_query` FROM `system`.`tables` │ 10848 │
2. │ SELECT DISTINCT arrayJoin([`database`, extract(`engine_full`, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) AS `database` FROM `system`.`tables` AS `tables` WHERE `engine` = 'Distributed' │ 6718 │
3. │ DESC TABLE system.tables │ 363 │
4. │ DESC TABLE system.functions │ 226 │
5. │ SELECT DISTINCT `name`, replaceRegexpOne(`create_query`, 'CREATE (FUNCTION)', 'CREATE \\1 IF NOT EXISTS') FROM `system`.`functions` AS `tables` WHERE `create_query` != '' │ 225 │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────┘
This caused the CPU on the ClickHouse to be overloaded and returned error TOO_MANY_SIMULTANEOUS_QUERIES .
Could you help me understand what could have caused this? Is there any config I can tune to make these queries less aggressive?
This is related to issue: https://github.com/Altinity/clickhouse-operator/issues/1685, because I think some of the high query load is caused by clusterAllReplicas used in some of these CHOP queries which amplified the load received by each ClickHouse node
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with related issue #1685 and the query examples in this report. Trace how CHOP reconciliation generates metadata queries, including the use of clusterAllReplicas, and identify any existing configuration that controls their frequency or concurrency. Done means the cause and a concrete mitigation are documented or implemented, with evidence that the query load is reduced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, go, kubernetes
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100