spring-cloud / spring-cloud/spring-cloud-gateway

Performance issue with micrometer and otel baggage

Open
#3,821 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feedback-provided
Dominant language
Java
Stars
4.9k
Forks
3.5k
Avg merge
20h 57m
Merged PRs (30d)
8

Description

Describe the bug
I'm attempting to add several OpenTelemetry baggage items to the current tracing context using Micrometer and OpenTelemetry in a spring-cloud-starter-gateway-server-webflux application, but this appears to be significantly impacting throughput. It degrades from ~108 RPS to ~51 RPS 😟

Before baggage:

Image

After baggage:

Image

I have followed the steps given in the micrometer documentation page, i.e. by using ReactorBaggage and ObservationAwareBaggageThreadLocalAccessor

see here - https://docs.micrometer.io/tracing/reference/configuring.html#_context_propagation_with_micrometer_tracing

Sample

id("org.springframework.boot") version "3.5.0"
id("io.spring.dependency-management") version "1.1.7"

extra["springCloudVersion"] = "2025.0.0"


implementation("io.micrometer:micrometer-tracing")
implementation("io.micrometer:micrometer-tracing-bridge-otel")

Registering ObservationAwareBaggageThreadLocalAccessor:

@Bean
    fun otelWebFilter(
        tracer: Tracer,
        registry: ObservationRegistry,
    ): OtelWebFilter {
        ContextRegistry.getInstance()
            .registerThreadLocalAccessor(ObservationAwareBaggageThreadLocalAccessor(registry, tracer))
        ContextRegistry.getInstance().registerThreadLocalAccessor(ObservationAwareSpanThreadLocalAccessor(tracer))
        return OtelWebFilter(tracer)
    }

ReactorBaggage with contextWrite:

class OtelWebFilter(private val tracer: Tracer) : WebFilter {

    private val logger = LoggerFactory.getLogger(OtelWebFilter::class.java)

    override fun filter(
        exchange: ServerWebExchange,
        chain: WebFilterChain
    ): Mono<Void> {
        val baggage = mapOf(
            "username" to exchange.request.headers.getFirst("X-Username").orEmpty(),
            "tenantId" to exchange.request.headers.getFirst("X-Tenant-Id").orEmpty(),
        )
        return chain.filter(exchange)
            .doOnSubscribe { logger.debug("Setting baggage for path: {}", exchange.request.path) }
            .doOnSuccess { logger.debug("Setting baggage completed") }
            .contextWrite(ReactorBaggage.append(baggage))
    }
}

Please look into it and let me know if you need further details

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the report with the Kotlin Spring Cloud Gateway sample, starting at the OtelWebFilter, ReactorBaggage.contextWrite call, and registered ObservationAware baggage accessors. Compare throughput with and without baggage and context propagation; done means the performance cause is isolated and a validated fix or clear finding is reported.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring, spring-boot
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.