cockroachdb / cockroachdb/cockroach
kvcoord: introduce DistSender circuit breakers for just the liveness range
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Currently, DistSender circuit breakers can only be turned on or off in their entirety. However, there are [scalability concerns](https://github.com/cockroachdb/cockroach/issues/121206) when turning on circuit breakers for all ranges. Whenever a circuit breaker is tripped we launch a probe in the background which entails launching a goroutine. This has the potential for causing goroutine spikes on the client if a node experiences a failure mode that causes circuit breakers to get tripped for a large number of replicas on it.
One of the main motivations for introducing DistSender circuit breakers was https://github.com/cockroachdb/cockroach/issues/80713. Until the scalability piece is de-risked, we should introduce a third mode of DistSender circuit breakers that only apply to replicas of the liveness range. That allows us to not regress on https://roachperf.crdb.dev/?filter=&view=failover%2Fliveness%2Fdeadlock&tab=gce while also side-stepping the scalability concerns.
Jira issue: CRDB-38208
Epic CRDB-39897
Contributor guide
Assessment
This issue has not been assessed yet.