AwsXrayPropagator propagates baggage unexpectedly

Component(s)

aws-xray-propagator

What happened?

Description

The AwsXrayPropagator injects baggage into outgoing AWS calls unexpectedly, even when no global baggage propagator has been configured. It delegates directly to W3CBaggagePropagator. This is surprising because it's called by TracingExecutionInterceptor in opentelemetry-java-instrumentation. Enabling that instrumentation causes baggage to be propagated to AWS. In our case, this caused request failures in production because in some circumstances the size of baggage caused message headers to exceed the size accepted by S3 and SNS. It bypassed our custom propagator that had extra safeguards to prevent sending baggage to 3rd parties and was installed as a global propagator instead of the baggage propagator.

Steps to Reproduce

build.gradle:

plugins {
    id 'java'
}

sourceCompatibility = JavaVersion.VERSION_21
targetCompatibility = JavaVersion.VERSION_21

repositories {
    mavenCentral()
}

configurations {
    javaagent
}

dependencies {
    // Replace with opentelemetry-javaagent:2.16.0 and the test will pass
    javaagent 'io.opentelemetry.javaagent:opentelemetry-javaagent:2.22.0'

    testImplementation platform('software.amazon.awssdk:bom:2.28.26')
    testImplementation 'software.amazon.awssdk:sts'
    testImplementation 'software.amazon.awssdk:apache-client'

    testImplementation 'io.opentelemetry:opentelemetry-api:1.44.0'
    testImplementation 'io.opentelemetry:opentelemetry-context:1.44.0'

    testImplementation 'org.wiremock:wiremock:3.10.0'
    testImplementation 'org.junit.jupiter:junit-jupiter:5.11.4'
    testImplementation 'org.assertj:assertj-core:3.27.7'
    testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}

test {
    useJUnitPlatform()

    jvmArgs "-javaagent:${configurations.javaagent.asPath}"

    // W3CBaggagePropagator is deliberately absent from otel.propagators.
    // The bug is that AwsXrayPropagator.inject() calls it unconditionally anyway.
    systemProperty 'otel.propagators', 'tracecontext'
    systemProperty 'otel.traces.exporter', 'none'
    systemProperty 'otel.metrics.exporter', 'none'
    systemProperty 'otel.logs.exporter', 'none'
    systemProperty 'otel.javaagent.logging', 'simple'

    testLogging {
        events 'passed', 'failed', 'skipped'
        showStandardStreams = true
        exceptionFormat = 'full'
    }
}

settings.gradle:

rootProject.name = 'otel-awssdk-baggage-leak-repro'

src/test/java/MinimalAwsBaggageTest.java:

import com.github.tomakehurst.wiremock.junit5.WireMockExtension;
import com.github.tomakehurst.wiremock.verification.LoggedRequest;
import io.opentelemetry.api.baggage.Baggage;
import io.opentelemetry.context.Scope;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.sts.StsClient;
import software.amazon.awssdk.services.sts.model.GetCallerIdentityRequest;

import java.net.URI;
import java.util.List;

import static com.github.tomakehurst.wiremock.client.WireMock.*;
import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig;
import static org.assertj.core.api.Assertions.assertThat;

class MinimalAwsBaggageTest {

    @RegisterExtension
    static final WireMockExtension wm = WireMockExtension.newInstance()
        .options(wireMockConfig().dynamicPort())
        .build();

    private static final String CANARY_VALUE = "incident-repro-canary-value";

    @Test
    void awsXrayPropagatorShouldNotInjectBaggage() {
        wm.stubFor(post(anyUrl()).willReturn(aResponse()
            .withStatus(200)
            .withHeader("Content-Type", "text/xml")
            .withBody("""
                <GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
                  <GetCallerIdentityResult>
                    <Arn>arn:aws:iam::123456789012:user/repro</Arn>
                    <UserId>AKID:repro</UserId>
                    <Account>123456789012</Account>
                  </GetCallerIdentityResult>
                  <ResponseMetadata>
                    <RequestId>deadbeef-dead-beef-dead-beefdeadbeef</RequestId>
                  </ResponseMetadata>
                </GetCallerIdentityResponse>
                """)));

        StsClient stsClient = StsClient.builder()
            .region(Region.EU_WEST_1)
            .endpointOverride(URI.create("http://localhost:" + wm.getPort()))
            .credentialsProvider(StaticCredentialsProvider.create(
                AwsBasicCredentials.create("AKIAZZZZZZZZZZZZZZZZ", "ZZZZZZZZZZZZZ")))
            .httpClient(ApacheHttpClient.builder().build())
            .build();

        Baggage canary = Baggage.current().toBuilder()
            .put("baggage-leak-canary", CANARY_VALUE)
            .build();
        try (Scope ignored = canary.makeCurrent()) {
            stsClient.getCallerIdentity(GetCallerIdentityRequest.builder().build());
        }

        List<LoggedRequest> requests = wm.findAll(postRequestedFor(anyUrl()));
        assertThat(requests).hasSize(1);
        LoggedRequest captured = requests.get(0);

        System.out.println("=== Headers on outgoing AWS API call ===");
        captured.getHeaders().all().forEach(h -> {
            String v = h.firstValue();
            System.out.println("  " + h.key() + ": " +
                (v.length() > 120 ? v.substring(0, 120) + "... (" + v.length() + " chars)" : v));
        });

        assertThat(captured.getHeader("baggage"))
            .as("BUG: baggage header should not be injected when no baggage"
                + "propagator is in in otel.propagators")
            .isNull();

    }
}

Expected Result

Test passes, as no baggage header is placed on outgoing request.

Actual Result

$ gradle build

> Task :test
[otel.javaagent 2026-08-05 14:28:29:421 +0100] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.22.0

Gradle Test Executor 24 STANDARD_ERROR
    SLF4J(W): No SLF4J providers were found.
    SLF4J(W): Defaulting to no-operation (NOP) logger implementation
    SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.

MinimalAwsBaggageTest > awsXrayPropagatorShouldNotInjectBaggage() STANDARD_OUT
    === Headers on outgoing AWS API call ===
      Host: localhost:64954
      amz-sdk-invocation-id: 6af2c4f9-3938-7186-b0a8-7efac1ae0f2d
      amz-sdk-request: attempt=1; max=4
      Authorization: AWS4-HMAC-SHA256 Credential=AKIAZZZZZZZZZZZZZZZZ/20260805/eu-west-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-i... (286 chars)
      baggage: baggage-leak-canary=incident-repro-canary-value
      Content-Type: application/x-www-form-urlencoded; charset=UTF-8
      User-Agent: aws-sdk-java/2.28.26 md/io#sync md/http#Apache ua/2.0 os/Mac_OS_X#26.5.1 lang/java#22.0.1 md/OpenJDK_64-Bit_Server_VM#22... (205 chars)
      x-amz-content-sha256: ab821ae955788b0e33ebd34c208442ccfc2d406e2edc5e7a39bd6458fbb4f843
      X-Amz-Date: 20260805T132832Z
      X-Amzn-Trace-Id: Root=1-387feb82-03172f987e2f6093d1894fe0;Parent=c317e834871a0e24;Sampled=1
      Content-Length: 43
      Connection: keep-alive

MinimalAwsBaggageTest > awsXrayPropagatorShouldNotInjectBaggage() FAILED
    org.opentest4j.AssertionFailedError: [BUG: baggage header should not be injected when no baggagepropagator is in in otel.propagators]
    expected: null
     but was: "baggage-leak-canary=incident-repro-canary-value"
        at app//MinimalAwsBaggageTest.awsXrayPropagatorShouldNotInjectBaggage(MinimalAwsBaggageTest.java:77)

1 test completed, 1 failed

> Task :test FAILED

[Incubating] Problems report is available at: file:///Users/AnnetteWilson/claude/otel-baggage-leak-2026-06/minimaltest/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///Users/AnnetteWilson/claude/otel-baggage-leak-2026-06/minimaltest/build/reports/tests/test/index.html

* Try:
> Run with --scan to get full insights.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.14.2/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD FAILED in 5s
3 actionable tasks: 1 executed, 2 up-to-date

Discussion

It appears that this behaviour was intended by PR #2147. It is mentioned in the PR description:

Propagate sampling information between instrumented services - specifically, the sampling rule in the root service is passed to all downstream services via the trace state AND baggage, such that statistics can be recorded meaningfully for boost to be triggered for the root service in a distributed system. Both trace state and baggage are used in case the user's configuration places a propagator that overrides one of these values in any service along the call chain.

and

Ensure baggage is propagated with any modifications in AwsXrayPropagator by using W3CBaggagePropagator.getInstance().inject(context.with(baggage), setter, carrier);

We found it surprising that baggage could be propagated in HTTP headers when we did not include the baggage propagator in the otel.propagators system property (nor the corresponding environment variable). It's possible this behaviour is necessary, but we feel it should at least be more clearly documented in the AWS instrumentation and should have a reasonable mechanism to opt-out.

We are currently filtering out the headers after they've been added to the request, but we're worried this is probably not the most robust way to do it.

We did not think baggage would even be used by AWS but we are using New Relic and not X-Ray for distributed tracing, so we were not aware of how it is used.

AI disclosure

I used Claude Sonnet 4.6 via the OpenCode TUI to produce a minimal test and further edited it myself for minimality. I wrote this bug report by hand.

Component version

1.51.0-alpha

Log output

No response

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}