cockroachdb / cockroachdb/cockroach

kvserver: queues enqueue follower replicas under leader leases, wasting merge queue throughput

Open
#174,617 1 comment 0 reactions 0 assignees View on GitHub
A-kv-distribution A-leader-leases C-bug O-agent O-support T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Under leader leases, every `needsLease` queue enqueues follower replicas, which are then popped and dropped without a trace.

The add-time filter in `baseQueue.maybeAdd` → `replicaCanBeProcessed` ([pkg/kv/kvserver/queue.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/queue.go)) only rejects a replica when it can positively see a valid lease owned by another store: `if st.IsValid() && !st.OwnedBy(repl.StoreID())`. But a follower of a healthy leader-lease range can never prove the remote lease valid: once the lease's `MinExpiration` has passed, validity is defined by raft leader support, which only the leader can evaluate, so `leases.Status` ([pkg/kv/kvserver/leases/status.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/leases/status.go), the `LeaseLeader` not-owned-locally branch) returns `LeaseState_ERROR` ("leader lease is not held locally, cannot determine validity") — and `IsValid()` is false, so the filter passes. Under epoch and expiration leases the follower could verify validity (via the liveness record or the timestamp), so this filter used to wor. Every enqueued follower entry is then popped and discarded at the pop-time lease check (`processReplica` → `redirectOnOrAcquireLease` → `NotLeaseHolderError` → `errMarkNotAcquirableLease` → `return nil`), incrementing no metric (not successes, failures, or purgatory) and logging only at V(3).

Jira issue: CRDB-67818

Contributor guide

Open the contributing guide

Research direction

Start in pkg/kv/kvserver/queue.go at baseQueue.maybeAdd and replicaCanBeProcessed, then trace lease behavior in pkg/kv/kvserver/leases/status.go. Follow processReplica through redirectOnOrAcquireLease and the NotLeaseHolderError path. Done means follower replicas under leader leases are filtered appropriately instead of being enqueued and discarded without accounting.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.