cockroachdb / cockroachdb/cockroach

kvcoord: separate read/write tracking for DistSender circuit breakers

Open
#120,195 3 comments 0 reactions 0 assignees View on GitHub
A-kv-client C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

In #118943 we added DistSender circuit breakers, which will prevent DistSenders getting stuck on nodes with stalled disks, and instead try a different leaseholder elsewhere. However, recovery time is higher than desired, at [20 seconds with epoch leases](https://roachperf.crdb.dev/?filter=&view=failover%2Fnon-system%2Fdisk-stall&tab=gce) and [16 seconds with expiration leases](https://roachperf.crdb.dev/?filter=&view=failover%2Fnon-system%2Fdisk-stall%2Flease%3Dexpiration&tab=gce).

This is probably because reads still succeed until the node loses its lease. If we track stalls/errors separately for writes and reads, we can trip the breaker soon after writes stall, which should reduce the recovery time by something like 3-6 seconds.

Jira issue: CRDB-36539

Epic CRDB-39897

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.