cockroachdb / cockroachdb/cockroach

mixedversion,roachtest: improve network partition strategy

Open
#154,547 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-foundations A-testing C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently our network partition mutator in mixed version allows any given node to be partitioned from one or more nodes in the cluster. While this still maintains quorum:

1. Leaders and leases on the partitioned node need adequate time to failover. If the leader/leaseholder is partitioned from its replicas, we will have unavailability during that period of time.
2. Under epoch leases, a leader leaseholder split [can lead to unavailability](https://cockroachlabs.atlassian.net/wiki/spaces/CKB/pages/4632379393) if the two are partitioned from each other. This is a known issue of epoch leases but is fixed by leader leases.

The latter is harder to fix (e.g. requires a node restart), so it may be preferred to only inject network partitions under leader leases. The former could be solved by:

1. (Under RF=3) Picking 2 random nodes in the cluster as "protected nodes" which will never be intentionally partitioned. We pin these nodes with: `constraints = '{"+node=n1": 1, "+node=n2": 1}'` and `voter_constraints = '{"+node=n1": 1, "+node=n2": 1}'` such that we always maintain liveness. This is a similar approach that kvnemisis uses.
2. Only allowing one node to node partition from happening. This should ensure the leader/leasholder is never partitioned from all replicas and we maintain liveness.
3. Adding [adequate](https://roachperf.crdb.dev/?filter=failover) retries/wait time when a network partition is being injected. This should give the cluster enough time to failover.

Jira issue: CRDB-54991

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.