OpenTelemetry Java agent + Micrometer Tracing

Demo application is using OpenTelemetry Java agent and Micrometer for traces customization. In some cases these custom tags/attributes are getting lost. Not sure if this is a valid setup or if there is another recommended approach?

-javaagent:lib/opentelemetry-javaagent.jar

Micrometer dependencies are added via Gradle.

dependencies {
   implementation 'org.springframework.boot:spring-boot-starter-actuator'
   implementation 'org.springframework.boot:spring-boot-starter-webflux'
   implementation 'io.micrometer:micrometer-tracing-bridge-otel'
   runtimeOnly 'io.micrometer:micrometer-registry-prometheus'
   runtimeOnly 'io.projectreactor:reactor-core-micrometer'
   compileOnly 'org.projectlombok:lombok'
   annotationProcessor 'org.projectlombok:lombok'
   testImplementation 'org.springframework.boot:spring-boot-starter-test'
   testImplementation 'io.projectreactor:reactor-test'
}

A controller method does a WebFlux request to another endpoint – to simulate an external service call.
Both controller methods log the current span and attach the custom attribute the the span.

@RestController
@RequiredArgsConstructor
@Slf4j
public class WebFluxDemoController {

    private final WebClient.Builder builder;

    private final Tracer tracer;

    @GetMapping("/")
    public Mono<String> helloWorld() {
        return builder.baseUrl("http://localhost:8080").build()
                .get()
                .uri("/internal")
                .retrieve().bodyToMono(String.class)
                .handle((str, sink) -> {
                    tracer.currentSpan().tag("hello.world", str);
                    log.info("HelloWorld: Trace ID: {}, Span ID: {}",  tracer.currentSpan().context().traceId(),  tracer.currentSpan().context().spanId());
                    sink.next(str);
                });
    }

    @GetMapping("/internal")
    public Mono<String> helloWorldInternal() {
        return Mono
                .just("Hello World!")
                .handle((str, sink) -> {
                    tracer.currentSpan().tag("hello.world.internal", str);
                    log.info("HelloWorldInternal: Trace ID: {}, Span ID: {}",  tracer.currentSpan().context().traceId(),  tracer.currentSpan().context().spanId());
                    sink.next(str);
                });
    }
}

For the first controller method this is working perfectly. Span ID is matching the overall trace and custom attribute is set.

On the second call the added custom attribute is not displayed. The logged span ID seems to be connected to the caller method instead to the actual internal controller method 'helloWorldInternal'.
Same behavior occurs if using two separate Spring Boot applications.

Doing the same with Spring MVC instead of WebFlux provides the expected result. In both cases Spring Boot 3.0.1 and OpenTelemetry Java agent 1.22.0 is being used.

2023-01-13T23:50:41.959+01:00  INFO 61977 --- [           main] c.e.opentelemetry.WebFluxApplication     : Starting WebFluxApplication using Java 17.0.5 with PID 61977 (/Users/jch439/Development/git/playground/spring-opentelemetry/opentelemetry-webflux/build/classes/java/main started by jch439 in /Users/jch439/Development/git/playground/spring-opentelemetry)
2023-01-13T23:50:41.965+01:00  INFO 61977 --- [           main] c.e.opentelemetry.WebFluxApplication     : The following 1 profile is active: "main"
2023-01-13T23:50:42.717+01:00 ERROR 61977 --- [           main] i.n.r.d.DnsServerAddressStreamProviders  : Unable to load io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'. Use DEBUG level to see the full stack: java.lang.UnsatisfiedLinkError: failed to load the required native library
2023-01-13T23:50:42.960+01:00  INFO 61977 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
2023-01-13T23:50:43.229+01:00  INFO 61977 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8080
2023-01-13T23:50:43.239+01:00  INFO 61977 --- [           main] c.e.opentelemetry.WebFluxApplication     : Started WebFluxApplication in 1.425 seconds (process running for 2.459)
2023-01-13T23:50:55.404+01:00  INFO 61977 --- [ctor-http-nio-4] c.e.opentelemetry.WebFluxDemoController  : HelloWorldInternal: Trace ID: bc1c11fc981527732d70799403f6b843, Span ID: 8c3df081fc23f34c
2023-01-13T23:50:55.439+01:00  INFO 61977 --- [ctor-http-nio-3] c.e.opentelemetry.WebFluxDemoController  : HelloWorld: Trace ID: bc1c11fc981527732d70799403f6b843, Span ID: c8d872af11b76078

Start with the WebFluxDemoController methods helloWorld and helloWorldInternal and the Gradle dependencies shown in the report, then reproduce the trace with the OpenTelemetry Java agent. Compare the active span and custom attributes for the WebFlux and Spring MVC cases; done means the internal request uses its expected span and retains its custom attribute.