cockroachdb / cockroachdb/cockroach

kvserver/closedts: skip TestRPCConnUnblocksOnStopper under DRPC until stream multiplexing has flow control

Open
#171,592 0 comments 0 reactions 0 assignees View on GitHub
branch-master C-enhancement O-agent T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

`TestRPCConnUnblocksOnStopper` is skipped for the `useDRPC=true` variant. With DRPC stream multiplexing it intermittently OOMs (seen on CI under memory limits), and even when it passes it is no longer testing what it intends to.

### What the test assumes

The test verifies that stopper quiescence interrupts a blocked `stream.Send()`. Its design depends on `Send()` exerting backpressure once the peer stops reading:

- The receiver handler never reads from the stream (`pushUpdates` blocks until the test ends).
- The send loop and the test are held in lockstep by a buffered(1) channel: each `beforeSend` lets the test publish one more large update (10000 replicas), which the loop immediately sends.
- The test concludes the sender is blocked when `beforeSend` stays idle for 100ms.

Under gRPC this works: HTTP/2 flow control makes `Send()` block once the window fills, the 100ms idle gap is genuine backpressure, and memory stays bounded.

### Why DRPC misbehaves

DRPC stream multiplexing has no writer-side flow control. `MuxWriter.WriteFrame` appends each frame to an unbounded in-memory buffer and returns immediately, draining to the transport from a separate goroutine. So `Send()` never blocks at the writer.

When the peer stops reading, the server's per-stream receive ring buffer (256 slots) fills, its reader goroutine blocks, and it stops draining the socket. TCP backpressure then blocks the client's `MuxWriter` drain goroutine on the socket write, but the producer is unaffected and keeps appending. `mw.buf` grows without bound.

The test's only termination condition under DRPC is the 100ms idle gap, which can no longer be produced by real backpressure. (Hypothesis) It now depends on scheduling or GC jitter happening to stall the send loop for 100ms before the buffer exhausts memory. Whether a run passes or OOMs is therefore a race between that jitter and the memory limit, which is why it passes on machines with headroom and OOMs intermittently on memory-constrained CI.

### Why we skip rather than adapt

The scenario the test exercises is not reachable under DRPC today. Without writer-side flow control, `Send()` cannot reach a blocked state, so there is nothing for the stopper to interrupt, and any "pass" is a timing artifact rather than a verification of the intended behavior.

### Re-enabling

Re-enable the `useDRPC=true` variant once stream multiplexing has flow control. That work is tracked in [CRDB-62785](https://cockroachlabs.atlassian.net/browse/CRDB-62785).

Jira issue: CRDB-64716

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.