open-telemetry / open-telemetry/opentelemetry-java

MetricsView created with SdkMeterProviderUtil.appendFilteredBaggageAttributes adds high-cardinality attributes from http-server-metrics-state

Open
#7,134 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug
Dominant language
Java
Stars
2.5k
Forks
1k
Avg merge
3d 17h
Merged PRs (30d)
58

Description

Describe the bug
I'm creating an OTel Agent extension, and I want to include a low-cardinality Baggage Entry into some metrics (http.server.request.duration and others).

I followed this example https://github.com/open-telemetry/opentelemetry-java-instrumentation/discussions/7704#discussioncomment-4949461

In my agent extension, I have autoconfiguration code like:

  private SdkMeterProviderBuilder configureSdkMeterProvider(SdkMeterProviderBuilder meterProviderBuilder, ConfigProperties config) {

    ViewBuilder tenancyAttrBuilder = View.builder();
    SdkMeterProviderUtil.appendFilteredBaggageAttributes(
        tenancyAttrBuilder,
        name -> name.equals("my-tenancy-attribute");
    View tenancyAttrView = tenancyAttrBuilder.build();

    for (String metricName: ImmutableList.of("http.server.request.duration", "http.server.active_requests") {
      meterProviderBuilder.registerView(
          InstrumentSelector.builder().setName(metricName).build(),
          tenancyAttrView
      );
    }
    return meterProviderBuilder;
  }

What did you expect to see?
I expected the only new attribute that'd be added to the metrics would be "my-tenancy-attribute" from Baggage.

What did you see instead?
The attribute from baggage is added, but also added are a handful of high-cardinality attributes.
None of which match the predicate supplied to the View.

Stepping through the code, it seems like they get added on this line: https://github.com/open-telemetry/opentelemetry-java/blob/37969926d13f7f62cf4fa9b2f46794c01d0cfb3a/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/internal/view/AttributesProcessor.java#L200

Incoming is a collection of the attributes from Context's http-server-metrics-state

With either jetty or undertow autoinstrumentation, this results in a cardinality explosion due to entries being added for the following attributes:

  • url.query
  • url.path
  • user_agent.original
  • network.peer.port
  • network.peer.address
  • client.address

Messages are logged from the otel agent like:

[otel.javaagent 2025-02-21 03:53:44:316 +0000] [XNIO-1 task-18] WARN io.opentelemetry.sdk.metrics.internal.state.DefaultSynchronousMetricStorage - Instrument http.server.request.duration has exceeded the maximum allowed cardinality (1999).

What version and what artifacts are you using?
Artifacts: opentelemetry-java, opentelemetry-java-instrumentation. otel agent + my extension. otlp exporters.
Version: 1.47.0 of otel java, 2.13.1 of otel-java-instrumentation
How did you reference these artifacts?

root pom


   <properties>
        <opentelemetry-instrumentation-bom.version>2.13.1</opentelemetry-instrumentation-bom.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.opentelemetry.instrumentation</groupId>
                <artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>
                <version>${opentelemetry-instrumentation-bom.version}-alpha</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

agent extension pom:


    <dependencies>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-api</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry.semconv</groupId>
            <artifactId>opentelemetry-semconv</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry.semconv</groupId>
            <artifactId>opentelemetry-semconv-incubating</artifactId>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-sdk-extension-autoconfigure-spi</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-instrumentation-api</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry.javaagent</groupId>
            <artifactId>opentelemetry-javaagent-extension-api</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry.javaagent</groupId>
            <artifactId>opentelemetry-javaagent-tooling</artifactId>
            <scope>provided</scope>
        </dependency>
    </dependencies>

Environment
Compiler: (e.g., "Temurin 17.0.7")
OS: (e.g., "Ubuntu 20.04")
Runtime (if different from JDK above): (e.g., "Oracle JRE 8u251")
OS (if different from OS compiled on): (e.g., "Windows Server 2019")

❯ java -version
openjdk version "1.8.0_372"
OpenJDK Runtime Environment (Zulu 8.70.0.23-CA-macos-aarch64) (build 1.8.0_372-b07)
OpenJDK 64-Bit Server VM (Zulu 8.70.0.23-CA-macos-aarch64) (build 25.372-b07, mixed mode)

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/internal/view/AttributesProcessor.java around line 200 and trace how Context's http-server-metrics-state attributes reach the view. Reproduce with the listed agent versions and verify that the filtered view retains only the matching baggage attribute, without the reported high-cardinality HTTP attributes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.