cockroachdb / cockroachdb/cockroach

kvcoord: introduce DistSender circuit breakers for just the liveness range

Open
#123,117 0 comments 0 reactions 0 assignees View on GitHub
A-kv-client C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Currently, DistSender circuit breakers can only be turned on or off in their entirety. However, there are [scalability concerns](https://github.com/cockroachdb/cockroach/issues/121206) when turning on circuit breakers for all ranges. Whenever a circuit breaker is tripped we launch a probe in the background which entails launching a goroutine. This has the potential for causing goroutine spikes on the client if a node experiences a failure mode that causes circuit breakers to get tripped for a large number of replicas on it.

One of the main motivations for introducing DistSender circuit breakers was https://github.com/cockroachdb/cockroach/issues/80713. Until the scalability piece is de-risked, we should introduce a third mode of DistSender circuit breakers that only apply to replicas of the liveness range. That allows us to not regress on https://roachperf.crdb.dev/?filter=&view=failover%2Fliveness%2Fdeadlock&tab=gce while also side-stepping the scalability concerns.

Jira issue: CRDB-38208

Epic CRDB-39897

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.