cockroachdb / cockroachdb/cockroach

Improve network detection diagnostics to prove network partitions exist between nodes

Open
#95,378 1 comment 0 reactions 0 assignees View on GitHub
C-enhancement T-observability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
When a cluster is behaving badly due to networking issues, network partitions, or asymmetric (one-way) network partitions, etc those are currently hard to diagnose and prove while the cluster has an outage or is degraded.
If a cluster is down due to a network issue, you cannot use standard CRDB diagnostics because, by definition, the cluster is already down. We need a way to confirm if and where exactly the network issues are, external to anything that relies on the cluster being up and running.
The lack of this type of diagnostics has resulted in very long and drawn out troubleshooting scenarios and escalations between our CRL and customers.

**Describe the solution you'd like**
At the end of the day, the diagnostic tooling could prove or show evidence that there are in fact issues with networking between nodes. As suggested by @smcvey ideally the diagnostics will check RPC and SQL ports from node A to node B and vice versa. The diagnostic tooling should sit outside of CRDB, perhaps a CLI command or another external network detection tool, so that it is usable even when a cluster is down.

**Additional context**
This was inspired by an internal conversation ([Slack](https://cockroachlabs.slack.com/archives/CHKQGKYEM/p1671147396772919) link that is only accessible to CRL employees)
cc @mwang1026

Jira issue: CRDB-23492

Epic CRDB-32137

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.