cockroachdb / cockroachdb/cockroach
kvcoord: DistSender circuit breaker scalability
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We should make sure the DistSender circuit breakers work well at scale. The biggest risks are probably the memory overhead of creating one circuit breaker per active replica (is the GC policy sound?), and the goroutine overhead of launching a large number of probes.
We should run some teste. For example, how many circuit breakers do we create for a 200k range cluster with 50k replicas/node and a representative high-concurrency workload? How much memory do they use? How many probes do we launch if a node's disk stalls or it falls off the network? How many goroutines are launched? How does it impact workload latency? How soon do the probe goroutines shut down as the leases move and replicas go idle? This overlaps with #119918.
We should also look at the circuit breaker creation overhead when there is an influx of requests to previously unused replicas, for example during a cold start or following a large number of splits. This overlaps with #119919.
Jira issue: CRDB-37137
Contributor guide
Assessment
This issue has not been assessed yet.