cockroachdb / cockroachdb/cockroach

kvserver: (expiration|leader) leases allocation support

Open
#131,783 0 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The allocator uses NodeLiveness, indirectly via the StorePool to compute live and non-live replicas, as well as filtering stores as allocation targets.

Expiration and Leader Leases have the benefit that a range maintains availability as a function of the range's connectivity between replicas, rather than towards the liveness leaseholder, powering NodeLiveness.

The allocator isn't aware of this currently and will determine liveness information exclusively via the StorePool.

If there's a partial partition, expiration|leader leases allow a range to maintain quorum but if the liveness leaseholder is somewhere such that some replicas are considered unavailable/dead/suspect by the StorePool (using NodeLiveness and the liveness range) then the following undesirable situations will occur:

If **majority** of replica stores are considered unavailable/dead etc, the allocator will not act upon that range, even for membership reasons. Decommissioning is blocked, if one actually does die, allocator can't do anything etc.

https://github.com/cockroachdb/cockroach/blob/794f4f11aba06e85a487b3e153a0aa4d3ee4c70f/pkg/kv/kvserver/allocator/allocatorimpl/allocator.go#L1004-L1013

If a **minority** of replica stores are considered unavailable/dead etc then:
- if the leaseholder replica is on the side **partitioned from the liveness range leaseholder**, it will encounter the same problem mentioned in the majority case
- if the leaseholder replica is on the side **not partitioned from the liveness range leaseholder**, then it will eventually replace those that were (if possible), blocking any other actions at a lower priority, such as decommissioning, satisfying constraints, rebalancing etc. It will also be shifting a lot of data pointlessly, since they aren't actually dead. This will also place additional burden upon stores on the non-partitioned side, potentially overloading them as we shift every replica on the partitioned side over (again, if possible).

We should investigate these claims and the behavior generally, it shouldn't be too hard to test in roachprod.

Also, as a prerequisite to generally enabling either for all clusters at a version, we should address any hazards found.

Jira issue: CRDB-42699

Epic CRDB-56265

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.