spring-cloud / spring-cloud/spring-cloud-gateway

Unable to export Micrometer traces via OpenTelemetry - (Spring Boot 3.5.5, Spring Cloud 2025.0.0)

Open
#3,904 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

waiting-for-triage
Dominant language
Java
Stars
4.9k
Forks
3.5k
Avg merge
20h 57m
Merged PRs (30d)
8

Description

Issue
I am trying to trace my Spring Cloud Gateway application and have those traces exported via OpenTelemetry, however I have failed to get anything useful successfully exported.

I can see that my request has been given a trace id, and the trace is propagated to downstream services. My only issue is that Spring Cloud Gateway is not exporting the spans I would expect.

There have been similar issues raised (e.g. https://github.com/spring-cloud/spring-cloud-gateway/issues/3184) but I the solutions there do not seem to work for me. I am sure I am missing something; I just need some help identifying what that is.

Any help would be greatly appreciated!

Sample Application
I have created a small application below to reproduce this issue.

.
├── pom.xml
└── src
    └── main
        ├── java
        │   └── com
        │       └── example
        │           ├── CloudGatewayApplication.java
        │           └── CustomGatewayInterceptor.java
        └── resources
            └── application.yml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xmlns="http://maven.apache.org/POM/4.0.0"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.5.5</version>
    <relativePath /> <!-- Force parent lookup from Maven repositories -->
  </parent>

  <groupId>com.example</groupId>
  <artifactId>tracing-example</artifactId>
  <version>0.0.1</version>
  <packaging>jar</packaging>
  <name>Tracing Example</name>

  <dependencyManagement>
    <dependencies>
      <!-- Spring Cloud bill of materials -->
      <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-dependencies</artifactId>
        <version>2025.0.0</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>
    <!-- Spring Boot dependencies -->
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <!-- Spring Cloud dependencies -->
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-gateway-server-webflux</artifactId>
    </dependency>
    <!-- Tracing dependencies -->
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-tracing-bridge-otel</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-exporter-otlp</artifactId>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <!-- SpringFramework - Creates an executable jar -->
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
      </plugin>
    </plugins>
  </build>
</project>

package com.example;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class CloudGatewayApplication {

    public static void main(final String[] args) {
        SpringApplication.run(CloudGatewayApplication.class, args);
    }
}

package com.example;

import io.micrometer.observation.annotation.Observed;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.springframework.cloud.gateway.filter.GatewayFilterChain;
import org.springframework.cloud.gateway.filter.GlobalFilter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.Ordered;
import org.springframework.web.server.ServerWebExchange;
import reactor.core.publisher.Mono;


@Configuration
public class CustomGatewayInterceptor {

    @Bean
    public GlobalFilter customFilter() {
        return new CustomGlobalFilter();
    }

    public class CustomGlobalFilter implements GlobalFilter, Ordered {

        private static final Logger logger = LogManager.getLogger(CustomGlobalFilter.class);

        @Override
        @Observed(name = "com.example.CustomGlobalFilter", contextualName = "CustomGlobalFilter")
        public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
            logger.info("Inside CustomGlobalFilter");
            return chain.filter(exchange);
        }

        @Override
        public int getOrder() {
            return 1;
        }
    }
}

server:
  port: 8443
  http2:
    enabled: true
  ssl:
    enabled: false


spring:
  application:
    name: Test Tracing
  reactor:
    context-propagation: AUTO
  cloud:
    gateway:
      server:
        webflux:
          x-forwarded:
            enabled: false
          httpserver:
            wiretap: false
          routes:
            - id: TestRoute
              uri: https://echo.free.beeceptor.com
              predicates:
                - Path=/test/**
              filters:
                - RewritePath=/test/?(?<segment>.*), /$\{segment}

logging:
  level:
    io.micrometer: DEBUG
    io.opentelemetry: DEBUG

management:
  tracing:
    enabled: true
    propagation:
      type: w3c
    sampling:
      probability: 1
  otlp:
    tracing:
      endpoint: http://127.0.0.1:4318/v1/traces

OTEL Collector
This is the OTEL Collector configuration I am running via docker run -it --rm -p 4318:4318 -v $(pwd)/otel-collector.yml:/otel-collector.yml:ro otel/opentelemetry-collector --config=/otel-collector.yml.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:
    verbosity: normal

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]

Results
I invoke the gateway using curl, which results in the following logs, showing that the custom filter is being executed, but none of the tracing is exported to OpenTelemetry.

You can see that:

  • The incoming request from curl didn't have any trace details, so a new trace id 6c2cfbb7931efae19359ef15872f53e0 was generated within Spring Cloud Gateway.
  • The custom global filter was executed and the trace details ware included on the log printed (6c2cfbb7931efae19359ef15872f53e0-b1a77d955987ef1b).
  • The trace details were propagated to the downstream service correctly in the header Traceparent: 00-6c2cfbb7931efae19359ef15872f53e0-7f2a73ad786de5e6-01.
  • The tracing within my custom global filter does not show in the OTEL debug logs.
$ curl http://localhost:8443/test/tracing

{
  "method": "GET",
  "protocol": "https",
  "host": "echo.free.beeceptor.com",
  "path": "/tracing",
  "ip": "91.237.176.102:60509",
  "headers": {
    "Host": "echo.free.beeceptor.com",
    "User-Agent": "curl/7.61.1",
    "Accept": "*/*",
    "Traceparent": "00-6c2cfbb7931efae19359ef15872f53e0-7f2a73ad786de5e6-01",
    "Via": "2.0 Caddy",
    "Accept-Encoding": "gzip"
  },
  "parsedQueryParams": {}
}

$ java -jar target/tracing-example-0.0.1.jar

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::                (v3.5.5)

2025-09-04T10:29:54.384+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] com.example.CloudGatewayApplication      : Starting CloudGatewayApplication v0.0.1 using Java 21.0.8 with PID 30669 (/ems/users/njames/tmp/CloudGatewayTrace/target/tracing-example-0.0.1.jar started by njames in /ems/users/njames/tmp/CloudGatewayTrace)
2025-09-04T10:29:54.393+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] com.example.CloudGatewayApplication      : No active profile set, falling back to 1 default profile: "default"
2025-09-04T10:29:57.689+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.cloud.context.scope.GenericScope     : BeanFactory id=2d96c5cc-560a-39cd-9cd6-7fe5d0343ad3
2025-09-04T10:30:00.460+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [After]
2025-09-04T10:30:00.461+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Before]
2025-09-04T10:30:00.461+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Between]
2025-09-04T10:30:00.461+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Cookie]
2025-09-04T10:30:00.462+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Header]
2025-09-04T10:30:00.462+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Host]
2025-09-04T10:30:00.462+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Method]
2025-09-04T10:30:00.463+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Path]
2025-09-04T10:30:00.463+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Query]
2025-09-04T10:30:00.463+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [ReadBody]
2025-09-04T10:30:00.464+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [RemoteAddr]
2025-09-04T10:30:00.464+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [XForwardedRemoteAddr]
2025-09-04T10:30:00.464+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Weight]
2025-09-04T10:30:00.465+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [CloudFoundryRouteService]
2025-09-04T10:30:01.232+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint beneath base path '/actuator'
2025-09-04T10:30:02.018+01:00 DEBUG 30669 --- [Test Tracing] [           main] [                                                 ] i.o.e.internal.http.HttpExporterBuilder  : Using HttpSender: io.opentelemetry.exporter.sender.okhttp.internal.OkHttpHttpSender
2025-09-04T10:30:02.161+01:00 DEBUG 30669 --- [Test Tracing] [           main] [                                                 ] i.o.sdk.internal.JavaVersionSpecific     : Using the APIs optimized for: Java 9+
2025-09-04T10:30:02.871+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8443 (http)
2025-09-04T10:30:03.088+01:00  INFO 30669 --- [Test Tracing] [           main] [                                                 ] com.example.CloudGatewayApplication      : Started CloudGatewayApplication in 9.971 seconds (process running for 11.597)
2025-09-04T10:32:47.166+01:00 DEBUG 30669 --- [Test Tracing] [or-http-epoll-2] [                                                 ] i.m.t.o.p.BaggageTextMapPropagator       : Will propagate new baggage context for entries {}
2025-09-04T10:32:47.265+01:00  INFO 30669 --- [Test Tracing] [or-http-epoll-2] [6c2cfbb7931efae19359ef15872f53e0-b1a77d955987ef1b] c.e.C.CustomGlobalFilter                 : Inside CustomGlobalFilter

$ docker run -it --rm -p 4318:4318 -v $(pwd)/otel-collector.yml:/otel-collector.yml:ro otel/opentelemetry-collector --config=/otel-collector.yml

2025-09-04T09:31:35.671Z        info    service@v0.134.0/service.go:211 Starting otelcol...     {"resource": {"service.instance.id": "edc91222-b16d-4987-a9d9-405526bfcbef", "service.name": "otelcol", "service.version": "0.134.0"}, "Version": "0.134.0", "NumCPU": 4}
2025-09-04T09:31:35.671Z        info    extensions/extensions.go:41     Starting extensions...  {"resource": {"service.instance.id": "edc91222-b16d-4987-a9d9-405526bfcbef", "service.name": "otelcol", "service.version": "0.134.0"}}
2025-09-04T09:31:35.672Z        info    otlpreceiver@v0.134.0/otlp.go:179       Starting HTTP server    {"resource": {"service.instance.id": "edc91222-b16d-4987-a9d9-405526bfcbef", "service.name": "otelcol", "service.version": "0.134.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "[::]:4318"}
2025-09-04T09:31:35.673Z        info    service@v0.134.0/service.go:234 Everything is ready. Begin running and processing data. {"resource": {"service.instance.id": "edc91222-b16d-4987-a9d9-405526bfcbef", "service.name": "otelcol", "service.version": "0.134.0"}}
2025-09-04T09:32:52.434Z        info    Traces  {"resource": {"service.instance.id": "edc91222-b16d-4987-a9d9-405526bfcbef", "service.name": "otelcol", "service.version": "0.134.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 2, "spans": 2}
2025-09-04T09:32:52.436Z        info    ResourceTraces #0 service.name=Test Tracing telemetry.sdk.language=java telemetry.sdk.name=opentelemetry telemetry.sdk.version=1.49.0
ScopeTraces #0 org.springframework.boot@3.5.5
HTTP GET 6c2cfbb7931efae19359ef15872f53e0 7f2a73ad786de5e6 http.method=GET http.status_code=200 http.uri=http://localhost:8443/tracing spring.cloud.gateway.route.id=TestRoute spring.cloud.gateway.route.uri=https://echo.free.beeceptor.com:443
ResourceTraces #1 service.name=Test Tracing telemetry.sdk.language=java telemetry.sdk.name=opentelemetry telemetry.sdk.version=1.49.0
ScopeTraces #1 org.springframework.boot@3.5.5
http get 6c2cfbb7931efae19359ef15872f53e0 b1a77d955987ef1b exception=none http.url=/test/tracing method=GET outcome=SUCCESS status=200 uri=UNKNOWN
        {"resource": {"service.instance.id": "edc91222-b16d-4987-a9d9-405526bfcbef", "service.name": "otelcol", "service.version": "0.134.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces"}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the sample application's pom.xml, CloudGatewayApplication.java, CustomGatewayInterceptor.java, and application.yml, then review the provided OTEL Collector configuration and reproduction logs. Determine why the gateway's observed spans are not exported while trace propagation works. Done means the sample produces the expected gateway spans in the collector output, with the relevant behavior covered by an appropriate project test.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java, spring, spring-boot
Domain
api, backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.