cockroachdb / cockroachdb/cockroach
kvserver,kvcoord: requests stuck on a partitioned ex-leaseholder fail after 60s although the range recovered in seconds
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
While transferring a leader lease, the lease is temporarily converted to an expiration lease, and as such it's susceptible to the following partial partitioning scenario:
- The old leader/leaseholder has issued `TimeoutNow` to transfer the leadership to the new leaseholder.
- A partial partition cuts the new leaseholder off from most node but not all. The new leaseholder doesn't have quorum support, so it can't win an election, can't upgrade the lease to a leader lease, and also can't extend the expiration lease.
- A request reaches the new leaseholder (via proxy or directly), acquires latches, evaluates but it stuck in re-proposals.
- After 6s the new lease expires and one of the other nodes in the cluster picks it up, so new requests are unblocked.
- The requests parked on the temporary leaseholder only return after 60s, when the leaderless watcher or circuit breaker fires resulting in a `ReplicaUnavailableError` returned to the client.
Two independent improvements would have bounded the impact:
1. `kvserver`: proposals under a clock-expired lease aren't finished early. A replica can observe locally that its lease had expired, making its pending proposals unusable to their clients, but `refreshProposalsLocked` only reproposes (doesn't checks lease validity), so the stuck proposal and everything latched behind it waits out the 60s breaker/leaderless-watcher thresholds. Note the proposal can't simply be bounced with `NotLeaseHolderError`; it may have committed before the partition and can still apply, so it must be finished with an ambiguous error, which then releases the latch waiters to redirect.
2. `kvcoord`: proxied same-range RUEs skip `DistSender`'s retry logic. When the RUE returns, the `DistSender` would normally record it and try the remaining replicas (`sendToReplicas` explicitly handles a single replica's RUE not implying range unavailability), so a retry would have succeeded if another replica picked up the lease. Instead the error took the early return meant for RUEs leaked from a different range (e.g. via intent resolution), because a proxied response's RUE names the proxy target while `curReplica` is the proxy. Distinguishing on the RUE's `RangeID` would let proxied same-range RUEs fall into the normal retry path.
**Expected behavior**
A request stuck on a replica that has lost its lease should fail over or fail fast within seconds of the lease moving, not after the 60s unavailability thresholds; and the `DistSender` should not surface one replica's RUE while un-tried replicas remain.
**Additional context**
Found by kvnemesis (#173365); the test-side trigger was removed by adding lease preferences (cockroachlabs/cockroach#3539), so this tracks the product gaps. Same partial-partition family as #160140; #157966 shows the same server-side wedge surfacing as poisoned latches.
Jira issue: CRDB-66761
Contributor guide
Research direction
Start by tracing kvserver's refreshProposalsLocked handling of expired leases and kvcoord's sendToReplicas handling of same-range ReplicaUnavailableError responses. Use the partial-partition scenario described here and related issues #160140 and #157966 to understand the failure modes. Done means expired-lease proposals finish ambiguously so waiters can redirect, and DistSender retries eligible untried replicas instead of surfacing one replica's error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend-api-design, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100