ClickHouse / ClickHouse/ClickHouse

max_memory_usage_for_user MemoryTracker leaks during hanging mutations and OPTIMIZE queries

Open
#113,060 2 comments 0 reactions 0 assignees View on GitHub
comp-memory external potential bug
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

_No response_

### Describe what's wrong

ClickHouse version: 25.8.28.1 (official build)

During high IO activity some users hit per-user memory limit though no queries with high RAM utilization are running.

High IO activity is caused by sort of slow disks (cloud network disks) and OPTIMIZE or DELETE queries that stuck for more than 5000 seconds.

Per-user tracker vs. actual in-flight usage:

```
SELECT user, formatReadableSize(memory_usage) AS tracker, formatReadableSize(peak_memory_usage) AS peak
FROM system.user_processes;
┌─user──────────────────┬─tracker────┬─peak───────┐
│ airflow_unlimited │ 162.62 GiB │ 163.04 GiB │
│ airflow_dp_assisted │ 146.59 GiB │ 163.17 GiB │
│ default │ 0.00 B │ 0.00 B │
│ monitoring │ 0.00 B │ 0.00 B │
│ airflow_no_dp │ 0.00 B │ 0.00 B │
└───────────────────────┴────────────┴────────────┘
SELECT user, count() AS q, formatReadableSize(sum(memory_usage)) AS real_now
FROM system.processes GROUP BY user;
┌─user──────────────────┬─q─┬─real_now──┐
│ default │ 1 │ 7.61 KiB │
│ airflow_unlimited │ 3 │ 1.17 GiB │
│ airflow_dp_assisted │ 6 │ 84.24 MiB │
└───────────────────────┴───┴───────────┘
```
Global tracker vs. real RSS:

```
SELECT metric, formatReadableSize(value) FROM system.metrics WHERE metric = 'MemoryTracking';
┌─metric─────────┬─formatReadableSize(value)─┐
│ MemoryTracking │ 7.05 MiB │
└────────────────┴───────────────────────────┘
SELECT formatReadableSize(value) FROM system.asynchronous_metrics WHERE metric = 'MemoryResident';
┌─formatReadableSize(value)─┐
│ 23.02 GiB │
└───────────────────────────┘
```
Summary of the four numbers:

Per-user tracker airflow_unlimited (system.user_processes) | 162.62 GiB
-- | --
Actual current usage for that user (system.processes) | 1.17 GiB
Global tracker (MemoryTracking) | 7.05 MiB
Real process RSS (MemoryResident) | 23.02 GiB

So: global tracker (7 MiB) ≪ sum of per-user trackers (~309 GiB), and each per-user value is ~150× the user's real usage.

A trivial query (peak 3.2 GiB) by such a user is then killed:

```
Code: 241. DB::Exception: User memory limit exceeded: would use 162.98 GiB
(attempt to allocate chunk of 4.05 MiB bytes), maximum: 162.98 GiB.
OvercommitTracker decision: Query was selected to stop by OvercommitTracker
(162.98 GiB == max_memory_usage_for_user == 175000000000 bytes.)
```

Whenever we set max_memory_usage_for_user == 0 all queries run fine.

Did not test this behavior on latest release because we reproduce the issue on production environment and we do not want to upgrade ClickHouse on prod as soon as the new version is released (in case of any new bugs etc)

### Does it reproduce on the most recent release?

Yes

### How to reproduce

0) ClickHouse version: 25.8.28.1 (official build)
1) Use network cloud disks or event s3 disks
2) Set non-zero max_memory_usage_for_user limit for some user
3) Run some OPTIMIZE queries or some heavy mutations against large tables that would run for more than 5000 sec
4) Check system.processes and system.user_processes
5) Try to run some RAM utilizing queries (not necessary heavy ones)
6) Get MEMORY_LIMIT_EXCEEDED error
7) Set max_memory_usage_for_user == 0 and run the same queries again

### Expected behavior

Long running mutations or OPTIMIZE queries do not cause per-user memory limits to leak and their actual RAM usage is calculated during RAM utilization calculations

### Error message and/or stacktrace

_No response_

### Related issues and pull requests

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the discrepancy with system.processes, system.user_processes, system.metrics, and system.asynchronous_metrics while long-running OPTIMIZE or mutation queries run. Trace the per-user memory accounting for those queries and verify that completed or stalled work no longer inflates max_memory_usage_for_user; confirm that the reported tracker values match current usage and the reproduction no longer triggers MEMORY_LIMIT_EXCEEDED.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
backend, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.