open-telemetry / open-telemetry/opentelemetry-java-instrumentation
Kafka Streams interceptors instrumented missing CHILD-OF relationship receive -> send
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1.2k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 228
Description
I have a simple Kafka Streams API based application doing just the following processing:
builder.stream(this.topicIn, Consumed.with(Serdes.String(), Serdes.String()))
.mapValues(s -> s.toUpperCase())
.to(this.topicOut);
When it's instrumented by using the consumer and producer interceptors, I can see two spans: receive and send.
I expect the process span missing because it's something I should add somehow in the application when doing the mapValues I guess (which instead happens automatically by using the java agent).
The issue I see is that the two spans are completely unrelated, so send is not child of receive.
When using interceptors across two different applications (a producer and consumer), the receive is child of send as expected.
My thought is because of the following in the TracingConsumer implementation https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/TracingConsumerInterceptor.java#L26
It build and finish the span, linking to an existing one (in the case of a send -> receive) but doesn't take into account that the receive could be followed by a send (and maybe a process in the middle).
How do we should deal with this from a tracing point of view?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with TracingConsumerInterceptor.java at the referenced line and inspect how the Kafka consumer and producer interceptors create and finish spans. Reproduce the receive-to-send flow with the shown Kafka Streams pipeline and compare it with separate producer and consumer applications. Done means the relationship between the receive and send spans is defined and verified for the Kafka Streams case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- distributed-systems, observability, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100