kvcoord: improve DistSender circuit breaker recovery time

DistSender circuit breakers are intended to prevent clients from getting stuck on stalled leaseholders, e.g. following a disk stall (replacing the Pebble disk stall detector), deadlock, or partial partition where client timeouts fire before a NLHE is returned (due to lease acquisition delays). Even though the lease moves to a different node, clients can still get stuck, which manifests as persistent unavailability.

The `disk-stall` and `deadlock` variants of the `failover` tests show that they are effective at doing so, but the recovery times are far beyond the optimal target of 10 seconds: with epoch leases, disk stalls only recover after 20 seconds, while expiration leases recover after 16 seconds. With deadlocks, expiration leases recover much faster, after 9 seconds.

We should try to get the recovery time for `disk-stall` cases down to 10 seconds. Some options to explore:

* Map out the recovery timeline. Why does it take so long? Are there sequential timeouts being hit (e.g. client/network/circuit breaker/Raft/lease timeouts) that could run in parallel?

* Should the `LeaseInfo` probes be sensitive to disk stalls? Currently, they are served out of memory. However, the replica will lose its lease after the disk stalls: epoch lease liveness heartbeats do a sync disk write which will stall, and expiration lease extensions will fail. This probably explains some of the high disk stall recovery time, since we only start failing probes after the lease has expired.

* Consider tracking stall/error rates for reads and writes separately. With e.g. a disk stall, reads will likely be cached and succeed until the lease is lost, while writes will fail immediately. This would allow us to trip the breaker sooner, once writes go beyond the threshold. See #120195.

* We recently added a grace period before cancelling in-flight writes: #121074. This will likely increase recovery times. Is the 10 seconds grace period too long? Should we either reduce it, or subtract the probe threshold from it since the stall likely began several seconds before the probe launch?

* Are the probe thresholds and timeouts reasonable? Can/should we lower them?

Jira issue: CRDB-37139

Epic CRDB-39897