cockroachdb / cockroachdb/cockroach

sql: query-level disk usage stats miss spilling by planNode row containers

Open
#172,807 2 comments 0 reactions 0 assignees View on GitHub
A-sql-execution C-bug E-quick-win O-agent O-support P-3 T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

### Describe the problem

Several planNodes and the subquery-execution path buffer rows in a disk-backed row container via `rowContainerHelper` ([pkg/sql/buffer_util.go](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/buffer_util.go)): `bufferNode`/`scanBufferNode` ([buffer.go#L35](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/buffer.go#L35)), `applyJoinNode` ([apply_join.go#L120](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/apply_join.go#L120)), `recursiveCTENode` ([recursive_cte.go#L71-L73](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/recursive_cte.go#L71-L73)), materialized subquery results ([distsql_running.go#L2244](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/distsql_running.go#L2244)), set-returning routines ([routine.go#L350](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/routine.go#L350)), and session-scoped SQL cursors.

When such a container spills, the spill is invisible in query-level stats:

1. **`EXPLAIN ANALYZE`'s `max sql temp disk usage`** is computed from the flow disk monitors ([vectorized_flow.go#L498](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/colflow/vectorized_flow.go#L498)), which are children of the root temp-storage monitor ([server.go#L260](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/distsql/server.go#L260)). `rowContainerHelper.initMonitors` parents its disk monitor directly on `ServerConfig.ParentDiskMonitor` ([buffer_util.go#L113-L115](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/buffer_util.go#L113-L115)), a *sibling* of the flow disk monitor, so container spills are excluded from the total. The field can be absent entirely while the query spilled substantial amounts. An existing TODO notes this ([buffer_util.go#L88-L90](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/buffer_util.go#L88-L90)). The same understated value flows into `crdb_internal` statement statistics and the sampled-query log.
2. **`sql.distsql.queries.spilled`** is only incremented when a vectorized flow creates its temp directory ([vectorized_flow.go#L344](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/colflow/vectorized_flow.go#L344)), so a query whose only spill is a row container isn't counted. (`sql.disk.distsql.current/max` are attached to the root temp-storage monitor ([server_sql.go#L893](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/server/server_sql.go#L893)) and *do* observe these spills, node-wide.)

Row-exec processors that spill (hash joiner, sorter, windower, joinreader, inverted joiner) are **not** affected: their disk monitors are children of `flowCtx.DiskMonitor`.

The analogous *memory*-side attribution problem was fixed by 904e814a0657644164d183eb7af397f69e90a3f2: query-level `maximum memory usage` now consults the per-query exec monitor ([conn_executor_exec.go#L3307](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/conn_executor_exec.go#L3307)), which is an ancestor of the container's mem monitors. There is no disk-side analog; a per-query exec-level disk monitor that parents the container (and flow) disk monitors, sampled the same way, would likely fix both symptoms above.

Practical impact: for a query whose dominant disk consumer is a materialized CTE buffer, `EXPLAIN ANALYZE` reports a `max sql temp disk usage` that can drastically understate the true footprint (or omit the line entirely), which misdirects investigation. Seen in a recent support escalation where two ~240k-row CTE buffers were re-scanned 4x each and their spill was invisible. Today the only per-query evidence of a container spill is the `spilled to disk` trace event ([row_container.go#L660](https://github.com/cockroachdb/cockroach/blob/e3bff5d92ac171e3c45a0eb6cda5356b4182e4ed/pkg/sql/rowcontainer/row_container.go#L660)).

### To Reproduce

```sql
SET distsql_workmem = '2MiB';
CREATE TABLE t (k INT PRIMARY KEY, v STRING);
INSERT INTO t SELECT i, repeat('a', 100) FROM generate_series(1, 100000) AS g(i);
SET tracing = on;
EXPLAIN ANALYZE WITH w AS MATERIALIZED (SELECT * FROM t)
SELECT count(*) FROM w JOIN w AS w2 USING (k);
SET tracing = off;
-- Proof the spill happened:
SELECT message FROM [SHOW TRACE FOR SESSION] WHERE message LIKE '%spilled to disk%';
```

Verified on master (77e4c74dd40012ae97747ce09df0dbe1d89268d2): the trace query returns one spill event from the buffer's monitor (`spilled to disk: ... memory budget exceeded: 10240 bytes requested, 2088960 currently allocated`), yet the `EXPLAIN ANALYZE` output contains no `max sql temp disk usage` line at all (and by code inspection, `sql.distsql.queries.spilled` cannot increment on this path). Notably, query-level `maximum memory usage` *does* capture the buffer's memory via the exec monitor; only the disk side is missing.

### Expected behavior

Row-container spills are reflected in the query-level `max sql temp disk usage` field and in the spill metrics.

Per-operator attribution for these containers is tracked separately in #172808.

*Code references are permalinks to master @ e3bff5d92ac (2026-07-24).*

Jira issue: CRDB-66074

Contributor guide

Open the contributing guide

Research direction

Start with pkg/sql/buffer_util.go and the monitor setup, then compare it with query disk accounting in pkg/sql/colflow/vectorized_flow.go and query execution in pkg/sql/conn_executor_exec.go. Run the SQL reproduction and inspect EXPLAIN ANALYZE output, spill metrics, and trace events. Done means row-container spills appear in query-level disk usage and spill metrics without changing the separate per-operator attribution work.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, observability-sre
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.