cockroachdb / cockroachdb/cockroach
kvserver: queues enqueue follower replicas under leader leases
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Under leader leases, every queue that requires the leaseholder enqueues
follower replicas. Each entry is popped and dropped at the pop-time lease
check.
The add-time filter in `baseQueue.replicaCanBeProcessed` only rejects a
replica when it positively sees a valid lease owned by another store
(`st.IsValid() && !st.OwnedBy(repl.StoreID())`). A follower of a healthy
leader-lease range cannot produce that: past the lease's `MinExpiration`,
validity is defined by raft leader support, which only the leader can
evaluate, so `leases.Status` returns `LeaseState_ERROR` and `IsValid()` is
false. Epoch and expiration leases are verifiable by any replica, which is
why the filter worked before.
On clusters with many small ranges these entries crowd out the ones a store
can actually process. The merge queue is the most visible: it pins at its
size cap while spending nearly all of its pop budget on follower entries.
Jira issue: CRDB-67902
Contributor guide
Research direction
Start at baseQueue.replicaCanBeProcessed and trace the add-time lease filter, then compare it with the pop-time check and leases.Status behavior for leader leases. Inspect the merge queue's enqueue and pop paths to understand the follower-replica pressure. Done means follower replicas are not crowding queues that require the leaseholder, while eligible replicas remain processable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100