open-telemetry / open-telemetry/opentelemetry-java-instrumentation

feat(kafka): add messaging.kafka.cluster.id to producer and consumer spans

Open
#18,977 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement needs triage
Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 16h
Merged PRs (30d)
214

Description

Is your feature request related to a problem? Please describe.

When a service connects to multiple Kafka clusters, or when the same topic name exists across environments (dev/staging/prod), Kafka spans carry no information about which cluster a message came from. `messaging.kafka.cluster.id` is defined in the OTel messaging semantic conventions for Kafka but is not currently set by this instrumentation — making it impossible to distinguish spans by cluster from telemetry alone.

Describe the solution you'd like

Add `messaging.kafka.cluster.id` as a span attribute on both producer and consumer spans. The cluster ID is read from the client's own already-fetched `Metadata` object — no `AdminClient`, no extra broker connection, no additional security config. The attribute is emitted by default — no opt-in flag required — and cached per client instance after the first successful read.

Implementation handles Kafka 3.7+ correctly: `KafkaConsumer` became a thin delegate wrapper in 3.7 (KIP-848) and no longer holds `metadata` directly — the fix walks `KafkaConsumer.delegate → LegacyKafkaConsumer` to find the `metadata` field.

Cross-language effort

This feature is being added consistently across all four OTel language instrumentations:

Describe alternatives you've considered
  • Extracting from consumer `metrics()` tags — Kafka does not expose the cluster ID as a standard metric tag, so this is not reliable across versions.
  • Requiring users to configure the cluster ID manually — eliminates the zero-config goal.
Additional context

`messaging.kafka.cluster.id` is defined in the OTel semconv incubating attributes (KIP-78). The attribute is emitted by default, consistent with the other `Recommended` Kafka attributes this instrumentation already produces. A working implementation with staging verification is ready.

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding `+1` or `me too`, to help us triage it. Learn more here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the Java implementation referenced in pull request #18978, focusing on the Kafka producer and consumer spans, KafkaConsumer.delegate, LegacyKafkaConsumer, and the already-fetched Metadata object. Confirm that the cluster ID is added to both span types without an extra broker connection and that it is cached per client instance.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
distributed-systems, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.