Trace/span lost during stateful processing in Kafka streams

Discussed in https://github.com/open-telemetry/opentelemetry-java-instrumentation/discussions/6407

^{Originally posted by paulklos August 2, 2022}
Hi,

We have an application that consumes messages from a Kafka topic. It's configured with Kafka bindings using Spring Cloud Stream.

One of our processes uses does an aggregation on a KStream, which results in a KTable, which is then converted back into a KStream and picked up by the next processor.

The whole process consists of 4 processors.

The application is instrumented with opentelemetry-javaagent.jar, the latest version.

Up until the aggregation we get the same trace id in our log messages, and a different span id in each processor, which is very nice, and exactly what we would expect.

But after the aggregation the span and trace ids disappear. Inspection in the debugger of the current span shows only zeros.

Is this a known limitation, or is there something we can configure to have proper propagation?

I created a demo project to illustrate the issue. It can be found here.

The README describes the steps to take.

Start with the linked streams-instrumentation demo project and follow the reproduction steps in its README. Trace the KStream aggregation into the KTable, back to the KStream, and through the next processor using the opentelemetry-javaagent.jar setup. Done means determining whether trace and span context can be preserved after aggregation, or clearly documenting the limitation and configuration needed.