cockroachdb / cockroachdb/cockroach

kv: issue replication lag probe before lease transfer

Open
#96,304 3 comments 0 reactions 0 assignees View on GitHub
A-kv C-enhancement O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Problem**

In https://github.com/cockroachdb/cockroach/issues/81561, we outlined the risk of range unavailability due to lease transfers sent to lagging followers. Recall that a follower replica must apply a lease transfer through its Raft log before it can take over as a leaseholder. This means catching up on its log first, if it is behind. During the period where the incoming leaseholder is catching up on its log, the range is unavailable.

In that issue, we focused on followers that needed a Raft snapshot to catch up on their log. These situations result in high and unpredictable replication lag. Snapshots require a bulk transfer of data which is queued and paced.

However, there is a less severe form of the same issue. Lease transfers may still cause unavailability if the incoming leaseholder is connected to the leader's log (i.e. does not need a snapshot), but is trailing by many log entries. In general, any meaningful replication lag is a problem for lease transfers.

**Proposed solution**

To place a soft bound on the period of unavailability, we could first measure the replication lag of the prospective leaseholder before transferring it the lease. If the lag is too much, we could reject the lease transfer. One way to accomplish this would be to measure the duration of a fake write to be applied on the prospective leaseholder when pushed through Raft. We have most of the infrastructure needed to support this. See `WaitForApplication`.

Note that we cannot mandate that the follower is all the way caught up on its log. This could prevent followers that have a small but persistent lag due to network latency and steady traffic from ever qualifying for the lease, which is what we saw in https://github.com/cockroachdb/cockroach/issues/38065.

Jira issue: CRDB-24054

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing WaitForApplication infrastructure and the lease-transfer path; the issue names no implementation files or tests. Determine how a prospective leaseholder's replication lag can be measured before transfer and what threshold should reject it. Done means transfers avoid excessive catch-up unavailability without excluding followers with small persistent lag.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.