ClickHouse / ClickHouse/ClickHouse
max_memory_usage_for_user MemoryTracker leaks during hanging mutations and OPTIMIZE queries
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
_No response_
### Describe what's wrong
ClickHouse version: 25.8.28.1 (official build)
During high IO activity some users hit per-user memory limit though no queries with high RAM utilization are running.
High IO activity is caused by sort of slow disks (cloud network disks) and OPTIMIZE or DELETE queries that stuck for more than 5000 seconds.
Per-user tracker vs. actual in-flight usage:
```
SELECT user, formatReadableSize(memory_usage) AS tracker, formatReadableSize(peak_memory_usage) AS peak
FROM system.user_processes;
┌─user──────────────────┬─tracker────┬─peak───────┐
│ airflow_unlimited │ 162.62 GiB │ 163.04 GiB │
│ airflow_dp_assisted │ 146.59 GiB │ 163.17 GiB │
│ default │ 0.00 B │ 0.00 B │
│ monitoring │ 0.00 B │ 0.00 B │
│ airflow_no_dp │ 0.00 B │ 0.00 B │
└───────────────────────┴────────────┴────────────┘
SELECT user, count() AS q, formatReadableSize(sum(memory_usage)) AS real_now
FROM system.processes GROUP BY user;
┌─user──────────────────┬─q─┬─real_now──┐
│ default │ 1 │ 7.61 KiB │
│ airflow_unlimited │ 3 │ 1.17 GiB │
│ airflow_dp_assisted │ 6 │ 84.24 MiB │
└───────────────────────┴───┴───────────┘
```
Global tracker vs. real RSS:
```
SELECT metric, formatReadableSize(value) FROM system.metrics WHERE metric = 'MemoryTracking';
┌─metric─────────┬─formatReadableSize(value)─┐
│ MemoryTracking │ 7.05 MiB │
└────────────────┴───────────────────────────┘
SELECT formatReadableSize(value) FROM system.asynchronous_metrics WHERE metric = 'MemoryResident';
┌─formatReadableSize(value)─┐
│ 23.02 GiB │
└───────────────────────────┘
```
Summary of the four numbers:
Per-user tracker airflow_unlimited (system.user_processes) | 162.62 GiB
-- | --
Actual current usage for that user (system.processes) | 1.17 GiB
Global tracker (MemoryTracking) | 7.05 MiB
Real process RSS (MemoryResident) | 23.02 GiB
So: global tracker (7 MiB) ≪ sum of per-user trackers (~309 GiB), and each per-user value is ~150× the user's real usage.
A trivial query (peak 3.2 GiB) by such a user is then killed:
```
Code: 241. DB::Exception: User memory limit exceeded: would use 162.98 GiB
(attempt to allocate chunk of 4.05 MiB bytes), maximum: 162.98 GiB.
OvercommitTracker decision: Query was selected to stop by OvercommitTracker
(162.98 GiB == max_memory_usage_for_user == 175000000000 bytes.)
```
Whenever we set max_memory_usage_for_user == 0 all queries run fine.
Did not test this behavior on latest release because we reproduce the issue on production environment and we do not want to upgrade ClickHouse on prod as soon as the new version is released (in case of any new bugs etc)
### Does it reproduce on the most recent release?
Yes
### How to reproduce
0) ClickHouse version: 25.8.28.1 (official build)
1) Use network cloud disks or event s3 disks
2) Set non-zero max_memory_usage_for_user limit for some user
3) Run some OPTIMIZE queries or some heavy mutations against large tables that would run for more than 5000 sec
4) Check system.processes and system.user_processes
5) Try to run some RAM utilizing queries (not necessary heavy ones)
6) Get MEMORY_LIMIT_EXCEEDED error
7) Set max_memory_usage_for_user == 0 and run the same queries again
### Expected behavior
Long running mutations or OPTIMIZE queries do not cause per-user memory limits to leak and their actual RAM usage is calculated during RAM utilization calculations
### Error message and/or stacktrace
_No response_
### Related issues and pull requests
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reproducing the discrepancy with system.processes, system.user_processes, system.metrics, and system.asynchronous_metrics while long-running OPTIMIZE or mutation queries run. Trace the per-user memory accounting for those queries and verify that completed or stalled work no longer inflates max_memory_usage_for_user; confirm that the reported tracker values match current usage and the reproduction no longer triggers MEMORY_LIMIT_EXCEEDED.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100