cockroachdb / cockroachdb/cockroach
mixedversion,roachtest: improve network partition strategy
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Currently our network partition mutator in mixed version allows any given node to be partitioned from one or more nodes in the cluster. While this still maintains quorum:
1. Leaders and leases on the partitioned node need adequate time to failover. If the leader/leaseholder is partitioned from its replicas, we will have unavailability during that period of time.
2. Under epoch leases, a leader leaseholder split [can lead to unavailability](https://cockroachlabs.atlassian.net/wiki/spaces/CKB/pages/4632379393) if the two are partitioned from each other. This is a known issue of epoch leases but is fixed by leader leases.
The latter is harder to fix (e.g. requires a node restart), so it may be preferred to only inject network partitions under leader leases. The former could be solved by:
1. (Under RF=3) Picking 2 random nodes in the cluster as "protected nodes" which will never be intentionally partitioned. We pin these nodes with: `constraints = '{"+node=n1": 1, "+node=n2": 1}'` and `voter_constraints = '{"+node=n1": 1, "+node=n2": 1}'` such that we always maintain liveness. This is a similar approach that kvnemisis uses.
2. Only allowing one node to node partition from happening. This should ensure the leader/leasholder is never partitioned from all replicas and we maintain liveness.
3. Adding [adequate](https://roachperf.crdb.dev/?filter=failover) retries/wait time when a network partition is being injected. This should give the cluster enough time to failover.
Jira issue: CRDB-54991
Contributor guide
Assessment
This issue has not been assessed yet.