ClickHouse / ClickHouse/ClickHouse
A query-scope manager to control the spill actions of memory intensive processors
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
> (you don't have to strictly follow this form)
**Company or project name**
> Put your company name or project description here
**Use case**
Aggregation, join and sort are memory intensive operations. At present, their trigger the spill based on their own memory usage, unaware of the existence of others. So when there are aggregation, join and sort exists in the same query, it's easy to cause OOM. It's inconvenient to adjust the spill parameters for each processor. This problem has trobuled us for a long time.
**Describe the solution you'd like**
Could we have a central spill manager for each query to control the spill actions of each processor ? When the memory is running out, it calls the `spill` method of each spillable processor to spill data into external disk and release some memory.
**Describe alternatives you've considered**
> A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
> Add any other context or screenshots about the feature request here.
Contributor guide
Assessment
This issue has not been assessed yet.