open-telemetry / open-telemetry/opentelemetry-java
MetricsView created with SdkMeterProviderUtil.appendFilteredBaggageAttributes adds high-cardinality attributes from http-server-metrics-state
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2.5k
- Forks
- 1k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 58
Description
Describe the bug
I'm creating an OTel Agent extension, and I want to include a low-cardinality Baggage Entry into some metrics (http.server.request.duration and others).
I followed this example https://github.com/open-telemetry/opentelemetry-java-instrumentation/discussions/7704#discussioncomment-4949461
In my agent extension, I have autoconfiguration code like:
private SdkMeterProviderBuilder configureSdkMeterProvider(SdkMeterProviderBuilder meterProviderBuilder, ConfigProperties config) {
ViewBuilder tenancyAttrBuilder = View.builder();
SdkMeterProviderUtil.appendFilteredBaggageAttributes(
tenancyAttrBuilder,
name -> name.equals("my-tenancy-attribute");
View tenancyAttrView = tenancyAttrBuilder.build();
for (String metricName: ImmutableList.of("http.server.request.duration", "http.server.active_requests") {
meterProviderBuilder.registerView(
InstrumentSelector.builder().setName(metricName).build(),
tenancyAttrView
);
}
return meterProviderBuilder;
}
What did you expect to see?
I expected the only new attribute that'd be added to the metrics would be "my-tenancy-attribute" from Baggage.
What did you see instead?
The attribute from baggage is added, but also added are a handful of high-cardinality attributes.
None of which match the predicate supplied to the View.
Stepping through the code, it seems like they get added on this line: https://github.com/open-telemetry/opentelemetry-java/blob/37969926d13f7f62cf4fa9b2f46794c01d0cfb3a/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/internal/view/AttributesProcessor.java#L200
Incoming is a collection of the attributes from Context's http-server-metrics-state
With either jetty or undertow autoinstrumentation, this results in a cardinality explosion due to entries being added for the following attributes:
- url.query
- url.path
- user_agent.original
- network.peer.port
- network.peer.address
- client.address
Messages are logged from the otel agent like:
[otel.javaagent 2025-02-21 03:53:44:316 +0000] [XNIO-1 task-18] WARN io.opentelemetry.sdk.metrics.internal.state.DefaultSynchronousMetricStorage - Instrument http.server.request.duration has exceeded the maximum allowed cardinality (1999).
What version and what artifacts are you using?
Artifacts: opentelemetry-java, opentelemetry-java-instrumentation. otel agent + my extension. otlp exporters.
Version: 1.47.0 of otel java, 2.13.1 of otel-java-instrumentation
How did you reference these artifacts?
root pom
<properties>
<opentelemetry-instrumentation-bom.version>2.13.1</opentelemetry-instrumentation-bom.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>
<version>${opentelemetry-instrumentation-bom.version}-alpha</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
agent extension pom:
<dependencies>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.opentelemetry.semconv</groupId>
<artifactId>opentelemetry-semconv</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.opentelemetry.semconv</groupId>
<artifactId>opentelemetry-semconv-incubating</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-extension-autoconfigure-spi</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-instrumentation-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.opentelemetry.javaagent</groupId>
<artifactId>opentelemetry-javaagent-extension-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.opentelemetry.javaagent</groupId>
<artifactId>opentelemetry-javaagent-tooling</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
Environment
Compiler: (e.g., "Temurin 17.0.7")
OS: (e.g., "Ubuntu 20.04")
Runtime (if different from JDK above): (e.g., "Oracle JRE 8u251")
OS (if different from OS compiled on): (e.g., "Windows Server 2019")
❯ java -version
openjdk version "1.8.0_372"
OpenJDK Runtime Environment (Zulu 8.70.0.23-CA-macos-aarch64) (build 1.8.0_372-b07)
OpenJDK 64-Bit Server VM (Zulu 8.70.0.23-CA-macos-aarch64) (build 25.372-b07, mixed mode)
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/internal/view/AttributesProcessor.java around line 200 and trace how Context's http-server-metrics-state attributes reach the view. Reproduce with the listed agent versions and verify that the filtered view retains only the matching baggage attribute, without the reported high-cardinality HTTP attributes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100