cockroachdb / cockroachdb/cockroach

kvserver: raft leader should follow leaseholder, not vice versa

Open
#107,523 1 comment 0 reactions 0 assignees View on GitHub
C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

Currently, raft leadership "determines" (to a degree) where the lease lives and raft only supports leadership on replicas that have the most up-to-date log. In contrast, CRDB usually wants to determine where the lease is. As a result, we regularly struggle both in production and in testing with nudging the lease where it is needed and, under overload, keeping it there. Additionally, we have lots of complicated code that tries to make sure that the lease and leadership colocate (leadership tries to follow the lease but we don't allow acquiring lease on non-leaders most of the time... if that seems circular yes, it's complex!)

It would be beneficial to invert this relationship. We should be able to decide (having run appropriate health checks) that the lease is now on replica X and then this should happen (in situations where we just want the lease "somewhere", we would still prefer to do it on the raft leader, just since that's the most efficient place to do it). The main reasons we don't do it like that are explained in this mega comment:

https://github.com/cockroachdb/cockroach/blob/b2ba2e537141fb4f836c3ba18ed6238e41f916c4/pkg/kv/kvserver/replica_proposal_buf.go#L660-L797

In short, we're scared of submitting a lease that will take a long time to become visible on the new leaseholder if it is behind on the raft log. So we instead redirect to the raft leader, have it grab a lease, and then perform a lease transfer to the follower (which includes checking whether the follower is in good health, i.e. not behind).

**Describe the solution you'd like**

As we rework the lease protocol, we should bake these health checks into every lease acquisition in a way that avoids this roundabout complicated strategy.

**Describe alternatives you've considered**

**Additional context**

https://github.com/cockroachdb/cockroach/issues/105172

Jira issue: CRDB-30094

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.