open-telemetry / open-telemetry/opentelemetry-java-instrumentation

Kafka Streams interceptors instrumented missing CHILD-OF relationship receive -> send

Open
#6,444 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 18h
Merged PRs (30d)
228

Description

I have a simple Kafka Streams API based application doing just the following processing:

builder.stream(this.topicIn, Consumed.with(Serdes.String(), Serdes.String()))
                .mapValues(s -> s.toUpperCase())
                .to(this.topicOut);

When it's instrumented by using the consumer and producer interceptors, I can see two spans: receive and send.
I expect the process span missing because it's something I should add somehow in the application when doing the mapValues I guess (which instead happens automatically by using the java agent).
The issue I see is that the two spans are completely unrelated, so send is not child of receive.
When using interceptors across two different applications (a producer and consumer), the receive is child of send as expected.

My thought is because of the following in the TracingConsumer implementation https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/TracingConsumerInterceptor.java#L26

It build and finish the span, linking to an existing one (in the case of a send -> receive) but doesn't take into account that the receive could be followed by a send (and maybe a process in the middle).
How do we should deal with this from a tracing point of view?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with TracingConsumerInterceptor.java at the referenced line and inspect how the Kafka consumer and producer interceptors create and finish spans. Reproduce the receive-to-send flow with the shown Kafka Streams pipeline and compare it with separate producer and consumer applications. Done means the relationship between the receive and send spans is defined and verified for the Kafka Streams case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
distributed-systems, observability, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.