cockroachdb / cockroachdb/cockroach
colexec: hash aggregator spills to disk too eagerly in some cases
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Currently, the hash aggregator always buffers up to `hashAggregatorMaxBuffered` tuples (set to `coldata.MaxBatchSize = 4096`) from the input before it performs the aggregation on those tuples. If input tuples take up significant amount of memory, then we will spill to disk. We should consider making the hash aggregator behave dynamically under the memory pressure - e.g. if we see that we used up at least half of the available budget during the buffering step, then we could just proceed to aggregation. In some cases (e.g. if we only have `min` or alike functions) we might avoid spilling to disk altogether.
Note that at the moment of writing only about 40% of `distsql_workmem` is given to the hash aggregator (10% are put aside for the output batch and 50% are given to the spilling queue to support disk spilling).
Jira issue: CRDB-26485
Contributor guide
Assessment
This issue has not been assessed yet.