cockroachdb / cockroachdb/cockroach

kvserver: interleaving replication change and lease transfer leaves ranges in an intermediate state

Open
#106,294 0 comments 0 reactions 0 assignees View on GitHub
A-kv A-kv-test-failure-complex C-bug db-cy-23 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

The presence of `LEARNER` replicas or a joint config state -- `VOTER_INCOMING`+`VOTER_DEMOTING_LEARNER` is supposed to be transient and intermediate.

In #101414 we see a leaseholder transfer away its lease due to a concurrent drain or replication change. When this occurs, the outgoing leaseholder is unable to exit the joint config until the new leaseholder calls:

https://github.com/cockroachdb/cockroach/blob/042cb0038df9167780e2be1ac054b2569700bb75/pkg/kv/kvserver/replica_command.go#L1274-L1276

The new leaseholder does this when:

- The replicate queue checks and processes the range (worst case 10 mins)
- Another config change is issued (split, merge, change replicas, relocate)

In #101414 we also saw that even after exiting the joint config, a learner remains which is not cleaned up until the test times out (1 min). This prevents other `ChangeReplicas` which add a replica to the learner store (for the range). Unlike the joint config, the learner will be cleaned up only by higher level config changes or the replicate queue.

It is not guaranteed that a config change will be issued, so in the worst case the range will remain in this state until the leaseholder replicate queue processes the replica.

**To Reproduce**

See #101414

Affects master - v22.2 (known)

**Expected behavior**

A range is able to perform any valid replication changes soon after a previous replication change, even if that replication change is interleaved with a lease transfer described above.

**Additional context**

Unable to issue `RELOCATE FROM .. TO ..` commands to add a voter.
Reduced survivability during long joint config.

Jira issue: CRDB-29490

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.