elastic / elastic/beats

[Metricbeat] Kafka partition / consumergroup fail when hosts is not an advertised broker address

Open
#52,928 0 comments 1 reaction 0 assignees View on GitHub
enhancement Metricbeat Team:Obs-InfraObs
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

## Summary

The Kafka `partition` and `consumergroup` metricsets connect successfully and fetch cluster metadata, then refuse to collect unless the configured `hosts` entry string-matches a broker in `advertised.listeners`.

Standard Kafka clients treat `hosts` as a bootstrap/seed address: they use it only to discover brokers, then talk to the advertised ones. Metricbeat does the extra match in order to decide which broker identity this instance is collecting for. There is no setting to skip it.

That match fails in three common layouts:

1. `hosts` is a single bootstrap address (load balancer, ingress, or router). That address is not any individual advertised broker.
2. `hosts` is a Kubernetes pod-name template (for example `${kubernetes.pod.name}:9771`) that expands to the advertised short name. The string match can succeed, but the connection to the short name fails at DNS resolution because that name is not in cluster DNS from the Agent.
3. `hosts` is a resolvable FQDN for a broker whose advertised listener is the short hostname. The Agent connects fine, but exact host:port equality against `advertised.listeners` fails, so collection stops.

Error:

```text
Error fetching data for metricset kafka.partition: error in connect: no advertised broker with address : found
```

The same error is returned for `kafka.consumergroup`.

## What happens

After `broker.Open` and a metadata fetch succeed, `Connect` looks up the configured address in the advertised broker list and errors if nothing matches:

https://github.com/elastic/beats/blob/main/metricbeat/module/kafka/broker.go#L133-L144

```go
meta, err := queryMetadataWithRetry(b.broker, b.cfg, nil)
// ...
other := finder.findBroker(brokerAddress(b.broker), meta.Brokers)
if other == nil { // no broker found
closeBroker(b.broker)
return fmt.Errorf("no advertised broker with address %v found", b.Addr())
}
```

The first comparison is exact `host:port` equality against advertised addresses:

https://github.com/elastic/beats/blob/main/metricbeat/module/kafka/broker.go#L432-L434

https://github.com/elastic/beats/blob/main/metricbeat/module/kafka/broker.go#L594-L598

Fallbacks after that compare against the **local** machine (local IPs, reverse-DNS of those IPs, local hostname) or against advertised addresses that are already IPs. They do not treat `broker-1.namespace.svc.cluster.local:9771` and advertised `broker-1:9771` as the same broker.

There is no config flag to disable this check.

This matching was previously described as by design (each Metricbeat instance identifies "its" broker so it does not scrape the whole cluster): https://github.com/elastic/beats/issues/34053#issuecomment-1352770691

The gap is that there is still no supported way to collect when the only reachable address is a bootstrap endpoint, or when the advertised name is not the name the Agent can resolve.

## Proof

TLS handshake and metadata fetch succeeded; collection then stopped on the match.

**1. Bootstrap / load-balancer host**

```yaml
hosts: ["kafka-bootstrap.example.com:443"]
```

Cluster metadata advertised individual brokers (different host and port), not the bootstrap address. Result:

```text
no advertised broker with address kafka-bootstrap.example.com:443 found
```

A normal Kafka client using the same bootstrap address can discover brokers and continue. Metricbeat cannot.

**2. Kubernetes pod-name template (still not enough)**

```yaml
hosts: ["${kubernetes.pod.name}:9771"]
```

The template expands to the same short name Kafka advertises (`broker-1:9771`), so the `findBroker` string check would pass, but the Agent cannot resolve `broker-1` from its network namespace, so the TCP connection to that host does not open. Adding an FQDN (case 3) fixes resolution but then breaks the string match. A DaemonSet + Kubernetes condition + `hosts` template does not get past both checks at once.

**3. Resolvable FQDN vs advertised short name**

- Advertised listener: `broker-1:9771`
- Configured `hosts`: `broker-1.kafka.svc.cluster.local:9771` (this name resolves; the short name does not)

Exact match fails (`broker-1.kafka.svc.cluster.local` != `broker-1`), same error as (1). Using the short name so the match would pass is not usable: the Agent cannot resolve `broker-1`.

That is a catch-22: the name that matches metadata does not resolve; the name that resolves does not match metadata.

Listing every advertised broker in `hosts` is also impractical when brokers are Kubernetes pods that scale and get new names.

The IP fallback only helps when advertised addresses are IPs or the Agent is on that broker's host; it does not equate an FQDN with a short hostname.

## Expected

`hosts` should be usable as a bootstrap/seed address, the way other Kafka clients work: connect, read metadata, collect without requiring that seed address (or an exact string equal to the advertised hostname) to appear in `advertised.listeners`.

Contributor guide

Open the contributing guide

Research direction

Start in metricbeat/module/kafka/broker.go, especially Connect and the findBroker comparisons around the referenced lines. Reproduce the bootstrap, Kubernetes short-name, and resolvable-FQDN cases, then trace the existing metadata and broker-selection flow. Done means partition and consumergroup can collect through a bootstrap address without requiring an exact advertised-host match.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
distributed-systems, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.