cockroachdb / cockroachdb/cockroach

server: add memory monitoring to status server fan-out RPCs

Open
#164,763 0 comments 0 reactions 0 assignees View on GitHub
A-cluster-observability C-enhancement T-observability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Summary

Status server fan-out RPCs (via `iterateNodes`) do not participate in memory accounting. Each fan-out aggregates responses from all nodes in-memory without any `mon.BytesMonitor` or memory limits, relying entirely on implicit bounds (e.g., response size limits, max concurrency).

The admin server has a `memMonitor` field, but it is explicitly unlimited:

```go
// TODO(knz): We do not limit memory usage by admin operations
// yet. Is this wise?
server.memMonitor = mon.NewUnlimitedMonitor(...)
```

## Affected RPCs

All RPCs using `iterateNodes` / `iterateNodesExt` in `pkg/server/status.go` are affected, including but not limited to:

- `ListContentionEvents`
- `ListDistSQLFlows`
- `ListExecutionInsights`
- `ListActiveSessionHistory`
- `Statements` (statement statistics)
- `HotRanges`
- `ListSessions`
- `SpanStats`
- `NetworkConnectivity`
- `Range` (range details)

## Current State

Individual RPCs use ad-hoc bounding (e.g., per-node response limits, pagination), but there is no unified memory accounting at the `iterateNodes` level. In large clusters, a fan-out that returns large responses from many nodes could cause memory pressure on the coordinating node.

## Proposed Approach

Add memory monitoring to the `iterateNodes` infrastructure itself rather than to individual RPCs. This could involve:

1. Passing a `mon.BytesMonitor` (or `mon.MemoryAccount`) into `iterateNodes` / `iterateNodesExt`.
2. Accounting for response sizes as they are aggregated.
3. Returning an error or applying backpressure when a memory threshold is exceeded.

This would provide a general solution for all fan-out RPCs without requiring each one to implement its own memory tracking.

## Context

This came up during review of #164440 (ASH sampling infrastructure). ASH already bounds memory structurally (fixed-size ring buffer, per-node response limits), but the broader pattern of unbounded fan-out aggregation applies to all status server RPCs.

Jira issue: CRDB-50283

Jira issue: CRDB-61008

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.