cockroachdb / cockroachdb/cockroach

kv: implement credit-based flow control for kvStreamer to prevent node overload

Open
#163,494 3 comments 0 reactions 0 assignees View on GitHub
C-performance O-support P-3 T-admission-control
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

### Background

A customer recently experienced an outage due to the kvStreamer fanning out too many Get requests, overloading receiving nodes. This was mitigated by setting `kv.streamer.concurrency_limit` to 128 (matching the default from an older CRDB version), but this static limit comes with performance costs.

### Problem

The current concurrency limit is static and doesn't adapt to:
- Cluster size changes
- Per-node load variations
- Individual query resource consumption

As noted in internal discussion: "If a node is generating a lot of requests for a range that's becoming overloaded, it would be better to queue those requests on the sender than the receiver."

### Proposed Solution: Credit-Based Flow Control

Implement a credit-based system where credits represent units of work one node can request from another:

**Global Credit Pool**
- Each node maintains a fixed global pool of credits
- On startup, allocate credits to each known peer node
- As cluster grows, per-node credit allocation decreases automatically
- Credits are returned when work completes (via RPC response)

**Per-Query Credit Limit**
- Each streamer/query has a maximum number of outstanding credits
- Prevents single queries from monopolizing the global pool
- Other queries can make progress even when one is credit-limited

**Implementation Options for kvStreamer**
Since kvStreamer controls fan-out, we have flexibility in credit allocation strategy:

1. **Prioritize completion**: Give streamers their full limit until global pool exhausted (process and free credits faster)
2. **Minimum guarantees**: Reserve minimum credits per streamer to avoid disrupting OLTP workloads
3. **Hybrid approaches**: Combine strategies based on query type or priority

### Benefits

- **Dynamic adaptation**: Automatically adjusts to cluster size
- **Load distribution**: Prevents sender from overwhelming receivers
- **Query isolation**: Per-query limits prevent resource monopolization
- **Backpressure**: Queues requests on sender rather than receiver

### Next Steps

1. Evaluate feasibility and performance characteristics
2. Design credit allocation and return protocol
3. Determine appropriate credit pool sizing
4. Implement per-query credit tracking
5. Add metrics for credit pool utilization

---

This issue tracks the exploration and potential implementation of this approach as a more dynamic alternative to static concurrency limits.

Jira issue: CRDB-60244

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.