ClickHouse / ClickHouse/ClickHouse

A query-scope manager to control the spill actions of memory intensive processors

Open
#72,663 5 comments 0 reactions 0 assignees View on GitHub
comp-query-execution external feature
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

> (you don't have to strictly follow this form)

**Company or project name**

> Put your company name or project description here

**Use case**

Aggregation, join and sort are memory intensive operations. At present, their trigger the spill based on their own memory usage, unaware of the existence of others. So when there are aggregation, join and sort exists in the same query, it's easy to cause OOM. It's inconvenient to adjust the spill parameters for each processor. This problem has trobuled us for a long time.

**Describe the solution you'd like**

Could we have a central spill manager for each query to control the spill actions of each processor ? When the memory is running out, it calls the `spill` method of each spillable processor to spill data into external disk and release some memory.

**Describe alternatives you've considered**

> A clear and concise description of any alternative solutions or features you've considered.

**Additional context**

> Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.