open-telemetry / open-telemetry/opentelemetry-java-instrumentation

Kafka instrumentation does not emit semconv messaging metrics (messaging.publish.duration / messaging.receive.duration)

Open
#19,094 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement needs triage
Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 18h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.

The Kafka client instrumentation emits producer/consumer spans but does not emit the semantic-convention messaging metrics messaging.publish.duration and messaging.receive.duration (+messaging.receive.messages).

The shared factory KafkaInstrumenterFactory (module instrumentation/kafka/kafka-clients/kafka-clients-common-0.11) never calls addOperationMetrics(...) on its producer/consumer instrumenters, so these metrics are never produced — and there is no configuration flag to turn them on today.

Note this is unrelated to the Kafka client native metrics (kafka.producer.* / kafka.consumer.*) bridged via OpenTelemetryMetricsReporter; those are the client's own metrics, not the portable semconv messaging metrics.

Describe the solution you'd like

Register the existing operation-metrics listeners on the Kafka instrumenters, mirroring the Pulsar instrumentation which already does this:

  • Producer instrumenter → MessagingProducerMetrics.get() → emits messaging.publish.duration
  • Consumer process/receive instrumenters → MessagingConsumerMetrics.get() → emits messaging.receive.duration and messaging.receive.messages

Both listeners already exist in instrumentation-api-incubator (io.opentelemetry.instrumentation.api.incubator.semconv.messaging.*). Because this is the shared factory, one change covers the javaagent, the kafka-clients-2.6 library (and therefore KafkaTelemetry / the Spring Boot starter), and spring-kafka.

Open question for maintainers: the messaging-metrics semconv is still experimental. Pulsar emits these unconditionally (precedent for default-on), but you may prefer gating them behind an experimental flag. Happy to follow whichever direction you prefer.

Describe alternatives you've considered
  • Do nothing / rely on native Kafka client metrics (kafka.producer.*): rejected — those are not the portable semconv messaging metrics and differ per messaging system.
  • Wire metrics in the Spring Boot starter / KafkaTelemetry: rejected — those only wire up the existing instrumentation. The metric set is decided in KafkaInstrumenterFactory, so the shared factory is the correct, high-leverage location.
Additional context

Precedent (reference implementation): instrumentation/pulsar/pulsar-2.8/javaagent/.../telemetry/PulsarSingletons.java already registers MessagingProducerMetrics / MessagingConsumerMetrics.

Semconv: https://github.com/open-telemetry/semantic-conventions/blob/main/docs/messaging/messaging-metrics.md

Verified the producer/consumer instrumenters in KafkaInstrumenterFactory have no addOperationMetrics(...) on main. Other span-only messaging instrumentations (JMS, SQS, RocketMQ) could get the same treatment, but this request is scoped to Kafka. I'm happy to open a PR.

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in instrumentation/kafka/kafka-clients/kafka-clients-common-0.11 at KafkaInstrumenterFactory, then compare the operation-metrics registration in the referenced PulsarSingletons.java implementation. Trace the producer and consumer instrumenters and existing MessagingProducerMetrics/MessagingConsumerMetrics listeners. Done means Kafka emits the requested publish and receive semantic-convention metrics, with the default-on or experimental-flag behavior resolved by maintainers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability-sre
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.