cockroachdb / cockroachdb/cockroach
Improve network detection diagnostics to prove network partitions exist between nodes
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
When a cluster is behaving badly due to networking issues, network partitions, or asymmetric (one-way) network partitions, etc those are currently hard to diagnose and prove while the cluster has an outage or is degraded.
If a cluster is down due to a network issue, you cannot use standard CRDB diagnostics because, by definition, the cluster is already down. We need a way to confirm if and where exactly the network issues are, external to anything that relies on the cluster being up and running.
The lack of this type of diagnostics has resulted in very long and drawn out troubleshooting scenarios and escalations between our CRL and customers.
**Describe the solution you'd like**
At the end of the day, the diagnostic tooling could prove or show evidence that there are in fact issues with networking between nodes. As suggested by @smcvey ideally the diagnostics will check RPC and SQL ports from node A to node B and vice versa. The diagnostic tooling should sit outside of CRDB, perhaps a CLI command or another external network detection tool, so that it is usable even when a cluster is down.
**Additional context**
This was inspired by an internal conversation ([Slack](https://cockroachlabs.slack.com/archives/CHKQGKYEM/p1671147396772919) link that is only accessible to CRL employees)
cc @mwang1026
Jira issue: CRDB-23492
Epic CRDB-32137
Contributor guide
Assessment
This issue has not been assessed yet.