open-telemetry / open-telemetry/opentelemetry-java-instrumentation
Metrics produced by instrumentation should have priority over metric bridge metrics
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1.2k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 214
Description
Issue Description
Metric bridges allow to capture existing metrics and convert them to OpenTelemetry metrics.
According to https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/15333, we have at least 3 metric bridges so far:
- kafka-clients-2.6
- micrometer-1.5
- dropwizard-metrics-4.0
The metrics captured through those bridges are not part of semantic conventions, nor do they follow the semantic conventions recommendations for metric names and attributes.
As far as I know, this is one of the reasons why micrometer metrics are disabled by default.
For micrometer metrics, they are very common with spring-boot applications, enabling them produces metrics that conflict with the stable JVM runtime metrics semconv.
The issue is the same for all bridges, but most particularly problematic with Micrometer, so I'll illustrate with Micrometer even if it applies to all metric bridges.
So from users perspective, this is not ideal because they have to choose between:
- do not use those micrometer metrics: otel provides equivalent ones, but this would not allow to use anything not yet covered by otel metrics nor any custom metric.
- implement filtering of conflicting metrics on the collector side
- deal with conflicts on the backend side
- modify or configure the application to filter those metrics
Of course, all of this will happen after finding that those extra conflicting/noisy metrics cause a problem downstream, which is not ideal.
For example, with a Spring Boot application with Micrometer, the jvm.memory.used (or jvm.memory.committed) metric is both produced by micrometer bridge and the instrumentation agent, it is also part of stable semantic conventions.
- the micrometer metric is mapped as a
Gaugeof typeDouble. - the runtime-metric instrumentation captures an
UpDownCounterof typeLong.
In this particular example the value of the metric remains the same, but the metric attributes are not, it could also happen with other metric properties like unit or type.
Proposal
The expected result should be that metrics produced by metric bridges should have lower priority over the metrics that are produced by instrumentation.
I see two possible ways to solve this:
- "simple": add a configuration option that has a list of all the metric names that should not be produced by metric bridges, the default value can be all the metrics that are part of semconv and/or produced by instrumentation, the obvious downside is having to maintain this list over time and making it verbose for users to extend/override it.
- "smart": implement that when metric bridges register/produce their metrics, if any metric exist don't create one, but we probably don't have any way to ensure proper ordering.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing where metric bridges register or produce metrics and where runtime-metric instrumentation emits jvm.memory.used and jvm.memory.committed. Compare registration ordering and metric properties, then choose and validate a priority rule that keeps instrumentation metrics while avoiding bridge conflicts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100