cockroachdb / cockroachdb/cockroach
kvcoord: separate read/write tracking for DistSender circuit breakers
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
In #118943 we added DistSender circuit breakers, which will prevent DistSenders getting stuck on nodes with stalled disks, and instead try a different leaseholder elsewhere. However, recovery time is higher than desired, at [20 seconds with epoch leases](https://roachperf.crdb.dev/?filter=&view=failover%2Fnon-system%2Fdisk-stall&tab=gce) and [16 seconds with expiration leases](https://roachperf.crdb.dev/?filter=&view=failover%2Fnon-system%2Fdisk-stall%2Flease%3Dexpiration&tab=gce).
This is probably because reads still succeed until the node loses its lease. If we track stalls/errors separately for writes and reads, we can trip the breaker soon after writes stall, which should reduce the recovery time by something like 3-6 seconds.
Jira issue: CRDB-36539
Epic CRDB-39897
Contributor guide
Assessment
This issue has not been assessed yet.