cockroachdb / cockroachdb/cockroach
sql: aggregate functions don't respect distsql_workmem limit
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
While working on addressing #79014, I realized that aggregate builtin functions often don't respect `distsql_workmem` limit. The problem is two-fold:
- in the vectorized ordered aggregator we always create an allocator with an unlimited monitor. The rationale for this is that we cannot spill to disk should the workmem budget be exceeded. We should re-evaluate this decision. We also might need to set `SingleDatumAggMemAccount` to an account bound to a limited monitor.
- for some row-by-row aggregate builtin functions (e.g. `array_agg`) we use the planner's monitor which doesn't know anything about `distsql_workmem`.
Note that this problem is not that important since we do still respect the global `--max-sql-memory` limit.
Jira issue: CRDB-26484
Contributor guide
Research direction
Start by tracing the vectorized ordered aggregator and row-by-row aggregate builtins such as `array_agg`, then inspect how `distsql_workmem` and the planner's monitor are connected. Done means aggregate functions in both paths respect the distsql work-memory limit while preserving the existing global `--max-sql-memory` protection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100