cockroachdb / cockroachdb/cockroach
kvserver/closedts: skip TestRPCConnUnblocksOnStopper under DRPC until stream multiplexing has flow control
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
`TestRPCConnUnblocksOnStopper` is skipped for the `useDRPC=true` variant. With DRPC stream multiplexing it intermittently OOMs (seen on CI under memory limits), and even when it passes it is no longer testing what it intends to.
### What the test assumes
The test verifies that stopper quiescence interrupts a blocked `stream.Send()`. Its design depends on `Send()` exerting backpressure once the peer stops reading:
- The receiver handler never reads from the stream (`pushUpdates` blocks until the test ends).
- The send loop and the test are held in lockstep by a buffered(1) channel: each `beforeSend` lets the test publish one more large update (10000 replicas), which the loop immediately sends.
- The test concludes the sender is blocked when `beforeSend` stays idle for 100ms.
Under gRPC this works: HTTP/2 flow control makes `Send()` block once the window fills, the 100ms idle gap is genuine backpressure, and memory stays bounded.
### Why DRPC misbehaves
DRPC stream multiplexing has no writer-side flow control. `MuxWriter.WriteFrame` appends each frame to an unbounded in-memory buffer and returns immediately, draining to the transport from a separate goroutine. So `Send()` never blocks at the writer.
When the peer stops reading, the server's per-stream receive ring buffer (256 slots) fills, its reader goroutine blocks, and it stops draining the socket. TCP backpressure then blocks the client's `MuxWriter` drain goroutine on the socket write, but the producer is unaffected and keeps appending. `mw.buf` grows without bound.
The test's only termination condition under DRPC is the 100ms idle gap, which can no longer be produced by real backpressure. (Hypothesis) It now depends on scheduling or GC jitter happening to stall the send loop for 100ms before the buffer exhausts memory. Whether a run passes or OOMs is therefore a race between that jitter and the memory limit, which is why it passes on machines with headroom and OOMs intermittently on memory-constrained CI.
### Why we skip rather than adapt
The scenario the test exercises is not reachable under DRPC today. Without writer-side flow control, `Send()` cannot reach a blocked state, so there is nothing for the stopper to interrupt, and any "pass" is a timing artifact rather than a verification of the intended behavior.
### Re-enabling
Re-enable the `useDRPC=true` variant once stream multiplexing has flow control. That work is tracked in [CRDB-62785](https://cockroachlabs.atlassian.net/browse/CRDB-62785).
Jira issue: CRDB-64716
Contributor guide
Assessment
This issue has not been assessed yet.