cockroachdb / cockroachdb/cockroach

kvserver: queues enqueue follower replicas under leader leases

Open
#174,686 1 comment 0 reactions 0 assignees View on GitHub
A-kv-distribution A-leader-leases C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Under leader leases, every queue that requires the leaseholder enqueues
follower replicas. Each entry is popped and dropped at the pop-time lease
check.

The add-time filter in `baseQueue.replicaCanBeProcessed` only rejects a
replica when it positively sees a valid lease owned by another store
(`st.IsValid() && !st.OwnedBy(repl.StoreID())`). A follower of a healthy
leader-lease range cannot produce that: past the lease's `MinExpiration`,
validity is defined by raft leader support, which only the leader can
evaluate, so `leases.Status` returns `LeaseState_ERROR` and `IsValid()` is
false. Epoch and expiration leases are verifiable by any replica, which is
why the filter worked before.

On clusters with many small ranges these entries crowd out the ones a store
can actually process. The merge queue is the most visible: it pins at its
size cap while spending nearly all of its pop budget on follower entries.

Jira issue: CRDB-67902

Contributor guide

Open the contributing guide

Research direction

Start at baseQueue.replicaCanBeProcessed and trace the add-time lease filter, then compare it with the pop-time check and leases.Status behavior for leader leases. Inspect the merge queue's enqueue and pop paths to understand the follower-replica pressure. Done means follower replicas are not crowding queues that require the leaseholder, while eligible replicas remain processable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.