cockroachdb / cockroachdb/cockroach

kvserver: lease transfer handshake protocol

Open
#116,231 2 comments 0 reactions 0 assignees View on GitHub
C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently, lease transfers are a one-way process: the outgoing leaseholder will give the lease to some other node it picks mostly based on local (possibly stale) information. However, the incoming leaseholder may not be able to actually take the lease, leading to unavailability. There's a range of reasons for this, e.g. being behind on replication, overload, partition, etc.

We have implemented several mitigations for this: checking the Raft leader state, liveness state, gossipping store IO overload, transferring expiration-based leases, etc.

Instead, we should use a handshake protocol for lease transfers. The outgoing leaseholder should submit a lease transfer proposal through Raft, and when the incoming leaseholder applies it it should perform any necessary local checks and then send an RPC/proposal back to the outgoing leaseholder to accept the transfer. The lease transfer proposal should only be valid for a limited time (e.g. a few seconds). This exercises the Raft machinery, makes sure the incoming leaseholder is caught up, and that it's ready to accept the lease.

This has significant overlap with https://github.com/cockroachdb/cockroach/issues/96304, and can replace many of our existing checks/mitigations.

Jira issue: CRDB-34529

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.