cockroachdb / cockroachdb/cockroach
server: add memory monitoring to status server fan-out RPCs
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Summary
Status server fan-out RPCs (via `iterateNodes`) do not participate in memory accounting. Each fan-out aggregates responses from all nodes in-memory without any `mon.BytesMonitor` or memory limits, relying entirely on implicit bounds (e.g., response size limits, max concurrency).
The admin server has a `memMonitor` field, but it is explicitly unlimited:
```go
// TODO(knz): We do not limit memory usage by admin operations
// yet. Is this wise?
server.memMonitor = mon.NewUnlimitedMonitor(...)
```
## Affected RPCs
All RPCs using `iterateNodes` / `iterateNodesExt` in `pkg/server/status.go` are affected, including but not limited to:
- `ListContentionEvents`
- `ListDistSQLFlows`
- `ListExecutionInsights`
- `ListActiveSessionHistory`
- `Statements` (statement statistics)
- `HotRanges`
- `ListSessions`
- `SpanStats`
- `NetworkConnectivity`
- `Range` (range details)
## Current State
Individual RPCs use ad-hoc bounding (e.g., per-node response limits, pagination), but there is no unified memory accounting at the `iterateNodes` level. In large clusters, a fan-out that returns large responses from many nodes could cause memory pressure on the coordinating node.
## Proposed Approach
Add memory monitoring to the `iterateNodes` infrastructure itself rather than to individual RPCs. This could involve:
1. Passing a `mon.BytesMonitor` (or `mon.MemoryAccount`) into `iterateNodes` / `iterateNodesExt`.
2. Accounting for response sizes as they are aggregated.
3. Returning an error or applying backpressure when a memory threshold is exceeded.
This would provide a general solution for all fan-out RPCs without requiring each one to implement its own memory tracking.
## Context
This came up during review of #164440 (ASH sampling infrastructure). ASH already bounds memory structurally (fixed-size ring buffer, per-node response limits), but the broader pattern of unbounded fan-out aggregation applies to all status server RPCs.
Jira issue: CRDB-50283
Jira issue: CRDB-61008
Contributor guide
Assessment
This issue has not been assessed yet.