open-telemetry / open-telemetry/opentelemetry-java-instrumentation

Multiple trace Id's generated from kafka consumer

Open
#2,531 17 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug repro provided
Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 18h
Merged PRs (30d)
228

Description

Describe the bug
We have deployed three services Service A, Service B and Service C. Service A interacts with Service B via API call and drops a message to Kafka Topic which is consumed by Service C. Service C consumes the record and calls external API call and inserts data into database. We observe multiple trace Id's generated for database and external API calls from kafka consumer instead of multiple span Id's.

Services Interaction: https://github.com/sunvuz/OpenTelemetry-poc/blob/30cdb74b4ac53bbba08fbc510e5900b3dc6bdf12/Kafka%20Consumer%20-New%20Traceid%20issue.PNG

Zipkin: Image with multiple trace Id's from consumer https://github.com/sunvuz/OpenTelemetry-poc/blob/30cdb74b4ac53bbba08fbc510e5900b3dc6bdf12/Kafka%20Consumer%20-New%20Traceid%20issue%20-Zipkin.PNG

Steps to reproduce
Sample Service's mimicking above behavior has been updated at : https://github.com/sunvuz/OpenTelemetry-poc

What did you expect to see?
Span Id for database and external API call Interaction instead of Trace Id's

What did you see instead?
New Trace Id's generated for database and external API interaction.

What version are you using?
v1.0.0 same is the behavior with 0.x.x versions

Environment
JDK 8+
OS : Linux

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked sample service and its Kafka consumer flow, then compare the trace and span behavior shown in the linked Zipkin image. Reproduce the database and external API interactions with version 1.0.0 or a 0.x.x version, and determine whether those interactions should remain under the consumer's trace. Done means the reported behavior is explained and corrected with verification of the resulting trace relationships.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.