cockroachdb / cockroachdb/cockroach

kvcoord: DistSender circuit breaker scalability

Open
#121,206 0 comments 0 reactions 0 assignees View on GitHub
A-kv-client C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We should make sure the DistSender circuit breakers work well at scale. The biggest risks are probably the memory overhead of creating one circuit breaker per active replica (is the GC policy sound?), and the goroutine overhead of launching a large number of probes.

We should run some teste. For example, how many circuit breakers do we create for a 200k range cluster with 50k replicas/node and a representative high-concurrency workload? How much memory do they use? How many probes do we launch if a node's disk stalls or it falls off the network? How many goroutines are launched? How does it impact workload latency? How soon do the probe goroutines shut down as the leases move and replicas go idle? This overlaps with #119918.

We should also look at the circuit breaker creation overhead when there is an influx of requests to previously unused replicas, for example during a cold start or following a large number of splits. This overlaps with #119919.

Jira issue: CRDB-37137

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.