cockroachdb / cockroachdb/cockroach
cdc: test Kafka network connectivity from nodes
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
Changefeed users may have network connectivity problems from CRDB to Kafka. This is difficult to troubleshoot. The error returned is often `kafka: client has run out of available brokers to talk to (Is your cluster reachable?)`, which I understand to result from any network problem.
Further, Cloud users do not have network troubleshooting tools (`ping`, `nslookup`, `traceroute`) because they can’t `ssh` into Cloud nodes. And, they don’t have access to logs, where they might `cat *.log | grep sarama`.
Problems might also include certs and auth.
**Describe the solution you'd like**
The ability to test Kafka connectivity from the CRDB process, via the SQL client. The Disaster Recovery team has implemented similar for cloud storage: https://github.com/cockroachdb/cockroach/pull/97517
The solution may be an extension of the above to accept `kafka://` URLs. We’d need some way to generate Kafka-specific workload.
See also kcat for inspiration: https://docs.confluent.io/platform/current/clients/kafkacat-usage.html
We might go further and add things like `WITH dns, traceroute, ping` etc, which would not be Kafka-specific. And then iterate on other sinks such as webhook and pubsub.
---
Possbile duplicate of https://github.com/cockroachdb/cockroach/issues/60178
Jira issue: CRDB-25245
Epic CRDB-6667
Contributor guide
Assessment
This issue has not been assessed yet.