open-telemetry / open-telemetry/opentelemetry-java-instrumentation
feat(kafka): add messaging.kafka.cluster.id to producer and consumer spans
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1.2k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 214
Description
Is your feature request related to a problem? Please describe.
When a service connects to multiple Kafka clusters, or when the same topic name exists across environments (dev/staging/prod), Kafka spans carry no information about which cluster a message came from. `messaging.kafka.cluster.id` is defined in the OTel messaging semantic conventions for Kafka but is not currently set by this instrumentation — making it impossible to distinguish spans by cluster from telemetry alone.
Describe the solution you'd like
Add `messaging.kafka.cluster.id` as a span attribute on both producer and consumer spans. The cluster ID is read from the client's own already-fetched `Metadata` object — no `AdminClient`, no extra broker connection, no additional security config. The attribute is emitted by default — no opt-in flag required — and cached per client instance after the first successful read.
Implementation handles Kafka 3.7+ correctly: `KafkaConsumer` became a thin delegate wrapper in 3.7 (KIP-848) and no longer holds `metadata` directly — the fix walks `KafkaConsumer.delegate → LegacyKafkaConsumer` to find the `metadata` field.
Cross-language effort
This feature is being added consistently across all four OTel language instrumentations:
- Java (this repo): #18978 — reads cluster id from already-fetched `Metadata`; no extra broker connection
- JavaScript: open-telemetry/opentelemetry-js-contrib#3577 — fetches via `admin().describeCluster()` once per lifecycle
- Python: open-telemetry/opentelemetry-python-contrib#4727 — reads from aiokafka/kafka-python/confluent-kafka client metadata
- .NET: open-telemetry/opentelemetry-dotnet-instrumentation#5206 — one-time background `AdminClient.DescribeClusterAsync()` per bootstrap-servers string
Describe alternatives you've considered
- Extracting from consumer `metrics()` tags — Kafka does not expose the cluster ID as a standard metric tag, so this is not reliable across versions.
- Requiring users to configure the cluster ID manually — eliminates the zero-config goal.
Additional context
`messaging.kafka.cluster.id` is defined in the OTel semconv incubating attributes (KIP-78). The attribute is emitted by default, consistent with the other `Recommended` Kafka attributes this instrumentation already produces. A working implementation with staging verification is ready.
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding `+1` or `me too`, to help us triage it. Learn more here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the Java implementation referenced in pull request #18978, focusing on the Kafka producer and consumer spans, KafkaConsumer.delegate, LegacyKafkaConsumer, and the already-fetched Metadata object. Confirm that the cluster ID is added to both span types without an extra broker connection and that it is cached per client instance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100