open-telemetry / open-telemetry/opentelemetry-java-instrumentation
Wrong span parent due to WithSpan when using java-agent with Ktor server and client
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1.2k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 228
Description
Describe the bug
I have stumbled upon a weird bug. The setup is this:
- A Ktor server, version 3.3.0
- The server is running with opentelemetry-javaagent version 2.20.0
- A route is calling a third party API endpoint with a Ktor client. We call this client instance "client A".
- The third party API requires a bearer token. To fetch the token we use the Auth plugin and fetch the token with a second client, called "client B".
- There is also a method annotated with
@WithSpan:fetchCountriesReturnStatus()
With this relatively common configuration, we observe the following issue: Whenever a request (using client A) triggers fetching a new token (using client B), that request's span has the wrong parent set. Please see the screenshot at the bottom of this post. The POST span is a token request triggered by the Auth plugin, using client B. The GET span is the span for the actual request the server wants to make, using client A. Note that the GET span is placed directly under the GET / span. It should be under the fetchCountriesReturnState span. I have not been able to pinpoint the exact reason this happens, but it is fairly consistent.
Steps to reproduce
I have made a reproducible case in a separate GitHub repository here. Instructions are provided in the README.
Expected behavior
The GET span in the screenshot should be nested inside the fetchCountriesReturnState span.
Actual behavior
The GET span for the request in the screenshot is placed directly under the root level GET / span.
Javaagent or library instrumentation version
opentelemetry-javaagent version 2.20.0
opentelemetry-instrumentation-annotations version 2.20.0
Environment
JDK: openjdk 17.0.16 2025-07-15
OS: MacOS Sequoia 15.6.1
Additional context
No response
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the README and reproducible case in the linked ktor-otel-bug-repro repository, then trace the Ktor server/client setup involving client A, client B, the Auth plugin, and fetchCountriesReturnStatus(). Reproduce the token-refresh request and inspect the java-agent and @WithSpan instrumentation. Done means the client A GET span is nested under fetchCountriesReturnStatus() even when client B fetches a new token.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100