Altinity / Altinity/clickhouse-operator

High query load from CHOP

Open
#1,686 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

Sometimes when CHOP is reconciling, it would send very high query load to each pod in the cluster.

e.g.: Below shows the top 5 queries received in 2 minutes by one ClickHouse node in a cluster of 120, all come from the CHOP


SELECT
    query,
    count() AS cnt
FROM system.query_log
WHERE (event_time >= toDateTime('2025-04-16 23:46:00')) AND (event_time <= toDateTime('2025-04-16 23:48:00'))
GROUP BY query
ORDER BY cnt DESC
LIMIT 5

Query id: 94e7460f-36db-4f99-9837-dc3040a2b9f6

   ┌─query────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───cnt─┐
1. │ SELECT DISTINCT `database`, `name`, `create_table_query` FROM `system`.`tables`                                                                                                                  │ 10848 │
2. │ SELECT DISTINCT arrayJoin([`database`, extract(`engine_full`, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+')]) AS `database` FROM `system`.`tables` AS `tables` WHERE `engine` = 'Distributed' │  6718 │
3. │ DESC TABLE system.tables                                                                                                                                                                         │   363 │
4. │ DESC TABLE system.functions                                                                                                                                                                      │   226 │
5. │ SELECT DISTINCT `name`, replaceRegexpOne(`create_query`, 'CREATE (FUNCTION)', 'CREATE \\1 IF NOT EXISTS') FROM `system`.`functions` AS `tables` WHERE `create_query` != ''                       │   225 │
   └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────┘

This caused the CPU on the ClickHouse to be overloaded and returned error TOO_MANY_SIMULTANEOUS_QUERIES .

Could you help me understand what could have caused this? Is there any config I can tune to make these queries less aggressive?

This is related to issue: https://github.com/Altinity/clickhouse-operator/issues/1685, because I think some of the high query load is caused by clusterAllReplicas used in some of these CHOP queries which amplified the load received by each ClickHouse node

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with related issue #1685 and the query examples in this report. Trace how CHOP reconciliation generates metadata queries, including the use of clusterAllReplicas, and identify any existing configuration that controls their frequency or concurrency. Done means the cause and a concrete mitigation are documented or implemented, with evidence that the query load is reduced.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, go, kubernetes
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.