elastic / elastic/beats

Kafka publish failed with: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Open
#36,601 5 comments 0 reactions 0 assignees View on GitHub
request-discuss Stalled Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

Dear friends, need your help.
For last couple of days stuck with below error. we are trying to send application logs to kafka, below is the kafka output config. We want to use kerberose auth type. It would be really great if anyone helps me on this issue.
Note: Masked sensitive info
output.kafka:

hosts: ["xxxxxx:port", "xxx:9092", "xx:9092"]
topic: "topic-filebeat-k8s-c88-logging"
kerberos.enabled: true
kerberos.auth_type: keytab
kerberos.username: "xxxxx"
kerberos.keytab: "/usr/share/filebeat/xxxx.keytab"
kerberos.service_name: "kafka"
kerberos.config_path: "/usr/share/filebeat/krb5.conf"
kerberos.realm: "XXX.COM"

errors

![h](https://github.com/elastic/beats/assets/37332000/56761c1e-27ed-4954-bd06-cdc2f2e0a7b3)

Contributor guide

Open the contributing guide

Research direction

Start with the output.kafka configuration shown in the issue, checking broker reachability and the Kafka Kerberos inputs: keytab, krb5.conf, realm, and service name. Use the Beats/Filebeat Kafka output and Kafka client logs to distinguish connectivity from authentication; done means a reproducible cause and a validated configuration or documented blocker.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.