open-telemetry / open-telemetry/opentelemetry-java-instrumentation

Metrics produced by instrumentation should have priority over metric bridge metrics

Open
#15,451 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 16h
Merged PRs (30d)
214

Description

Issue Description

Metric bridges allow to capture existing metrics and convert them to OpenTelemetry metrics.

According to https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/15333, we have at least 3 metric bridges so far:

  • kafka-clients-2.6
  • micrometer-1.5
  • dropwizard-metrics-4.0

The metrics captured through those bridges are not part of semantic conventions, nor do they follow the semantic conventions recommendations for metric names and attributes.

As far as I know, this is one of the reasons why micrometer metrics are disabled by default.
For micrometer metrics, they are very common with spring-boot applications, enabling them produces metrics that conflict with the stable JVM runtime metrics semconv.

The issue is the same for all bridges, but most particularly problematic with Micrometer, so I'll illustrate with Micrometer even if it applies to all metric bridges.

So from users perspective, this is not ideal because they have to choose between:

  • do not use those micrometer metrics: otel provides equivalent ones, but this would not allow to use anything not yet covered by otel metrics nor any custom metric.
  • implement filtering of conflicting metrics on the collector side
  • deal with conflicts on the backend side
  • modify or configure the application to filter those metrics

Of course, all of this will happen after finding that those extra conflicting/noisy metrics cause a problem downstream, which is not ideal.

For example, with a Spring Boot application with Micrometer, the jvm.memory.used (or jvm.memory.committed) metric is both produced by micrometer bridge and the instrumentation agent, it is also part of stable semantic conventions.

  • the micrometer metric is mapped as a Gauge of type Double.
  • the runtime-metric instrumentation captures an UpDownCounter of type Long.

In this particular example the value of the metric remains the same, but the metric attributes are not, it could also happen with other metric properties like unit or type.

Proposal

The expected result should be that metrics produced by metric bridges should have lower priority over the metrics that are produced by instrumentation.

I see two possible ways to solve this:

  • "simple": add a configuration option that has a list of all the metric names that should not be produced by metric bridges, the default value can be all the metrics that are part of semconv and/or produced by instrumentation, the obvious downside is having to maintain this list over time and making it verbose for users to extend/override it.
  • "smart": implement that when metric bridges register/produce their metrics, if any metric exist don't create one, but we probably don't have any way to ensure proper ordering.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing where metric bridges register or produce metrics and where runtime-metric instrumentation emits jvm.memory.used and jvm.memory.committed. Compare registration ordering and metric properties, then choose and validate a priority rule that keeps instrumentation metrics while avoiding bridge conflicts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.