cockroachdb / cockroachdb/cockroach

sql: aggregate functions don't respect distsql_workmem limit

Open
#100,548 0 comments 0 reactions 0 assignees View on GitHub
C-cleanup T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

While working on addressing #79014, I realized that aggregate builtin functions often don't respect `distsql_workmem` limit. The problem is two-fold:
- in the vectorized ordered aggregator we always create an allocator with an unlimited monitor. The rationale for this is that we cannot spill to disk should the workmem budget be exceeded. We should re-evaluate this decision. We also might need to set `SingleDatumAggMemAccount` to an account bound to a limited monitor.
- for some row-by-row aggregate builtin functions (e.g. `array_agg`) we use the planner's monitor which doesn't know anything about `distsql_workmem`.

Note that this problem is not that important since we do still respect the global `--max-sql-memory` limit.

Jira issue: CRDB-26484

Contributor guide

Open the contributing guide

Research direction

Start by tracing the vectorized ordered aggregator and row-by-row aggregate builtins such as `array_agg`, then inspect how `distsql_workmem` and the planner's monitor are connected. Done means aggregate functions in both paths respect the distsql work-memory limit while preserving the existing global `--max-sql-memory` protection.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.