cockroachdb / cockroachdb/cockroach
kvcoord: handle follower reads with DistSender circuit breakers
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The DistSender circuit breakers from https://github.com/cockroachdb/cockroach/pull/118943 should possibly handle follower reads better. The two main cases are:
* A replica partitioned away from the leader will fail to acquire a lease which may cause repeated client timeout errors, but it may still be able to serve follower reads below the closed timestamp.
* A stalled replica may not be able to serve follower reads either.
There are two interactions to keep in mind:
* Successful follower reads may mix with lease timeout errors, preventing the breaker from tripping and causing leased request failures.
* It may be possible for a replica with a tripped breaker to serve follower reads.
Potential follower reads use `RoutingPolicy_NEAREST` (this includes meta range lookups). We should consider adding specialized handling for these. `RoutingPolicy_LEASEHOLDER` may also be served as follower reads if the timestamp falls below the replica's closed timestamp, but this typically implies that the replica is in fact receiving closed timestamp updates from the leader -- it will typically take 6 seconds to trip the breaker, and the replica must have been failing at the start of this interval, implying that 6 second old follower reads will fail.
It may be reasonable to assume that the follower read timestamp is in the recent past, and that a faulty replica will remain faulty for longer than the follower read lag. Given that, we can likely omit successful follower reads when considering recent error runs, and trip the breaker regardless of successful follower reads. This will incur a latency penalty for follower reads that now have to hit other replicas, but once the follower read lag moves above the closed timestamp (typically a few seconds later) it will incur that penalty anyway.
Alternatively, we can differentiate handling of follower reads and other requests, and e.g. use separate breakers and probes for them, but it isn't clear that this is worthwhile.
Jira issue: CRDB-36391
Epic CRDB-39897
Contributor guide
Assessment
This issue has not been assessed yet.