cockroachdb / cockroachdb/cockroach

colexec: hash aggregator spills to disk too eagerly in some cases

Open
#100,549 1 comment 0 reactions 0 assignees View on GitHub
A-sql-vec C-cleanup T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently, the hash aggregator always buffers up to `hashAggregatorMaxBuffered` tuples (set to `coldata.MaxBatchSize = 4096`) from the input before it performs the aggregation on those tuples. If input tuples take up significant amount of memory, then we will spill to disk. We should consider making the hash aggregator behave dynamically under the memory pressure - e.g. if we see that we used up at least half of the available budget during the buffering step, then we could just proceed to aggregation. In some cases (e.g. if we only have `min` or alike functions) we might avoid spilling to disk altogether.

Note that at the moment of writing only about 40% of `distsql_workmem` is given to the hash aggregator (10% are put aside for the output batch and 50% are given to the spilling queue to support disk spilling).

Jira issue: CRDB-26485

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.