cockroachdb / cockroachdb/cockroach
kv: implement credit-based flow control for kvStreamer to prevent node overload
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
### Background
A customer recently experienced an outage due to the kvStreamer fanning out too many Get requests, overloading receiving nodes. This was mitigated by setting `kv.streamer.concurrency_limit` to 128 (matching the default from an older CRDB version), but this static limit comes with performance costs.
### Problem
The current concurrency limit is static and doesn't adapt to:
- Cluster size changes
- Per-node load variations
- Individual query resource consumption
As noted in internal discussion: "If a node is generating a lot of requests for a range that's becoming overloaded, it would be better to queue those requests on the sender than the receiver."
### Proposed Solution: Credit-Based Flow Control
Implement a credit-based system where credits represent units of work one node can request from another:
**Global Credit Pool**
- Each node maintains a fixed global pool of credits
- On startup, allocate credits to each known peer node
- As cluster grows, per-node credit allocation decreases automatically
- Credits are returned when work completes (via RPC response)
**Per-Query Credit Limit**
- Each streamer/query has a maximum number of outstanding credits
- Prevents single queries from monopolizing the global pool
- Other queries can make progress even when one is credit-limited
**Implementation Options for kvStreamer**
Since kvStreamer controls fan-out, we have flexibility in credit allocation strategy:
1. **Prioritize completion**: Give streamers their full limit until global pool exhausted (process and free credits faster)
2. **Minimum guarantees**: Reserve minimum credits per streamer to avoid disrupting OLTP workloads
3. **Hybrid approaches**: Combine strategies based on query type or priority
### Benefits
- **Dynamic adaptation**: Automatically adjusts to cluster size
- **Load distribution**: Prevents sender from overwhelming receivers
- **Query isolation**: Per-query limits prevent resource monopolization
- **Backpressure**: Queues requests on sender rather than receiver
### Next Steps
1. Evaluate feasibility and performance characteristics
2. Design credit allocation and return protocol
3. Determine appropriate credit pool sizing
4. Implement per-query credit tracking
5. Add metrics for credit pool utilization
---
This issue tracks the exploration and potential implementation of this approach as a more dynamic alternative to static concurrency limits.
Jira issue: CRDB-60244
Contributor guide
Assessment
This issue has not been assessed yet.