open-telemetry / open-telemetry/opentelemetry-java-instrumentation
HTTP Spans no longer emitted by AWS SDK v2 instrumentation
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1.2k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 228
Description
Describe the bug
Before v1.31*, calls using the AWS SDK would emit an HTTP Client Span. This span is now suppressed.
*I believe this behavior had existed since v1.28, based on my reading of this PR https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/8405/files#diff-ecc2a9e48ea1cac6e394aabfd4ece49b63cef5f08e950346e5e74cffd3c101e9
Steps to reproduce
- Set up the auto-instrumentation on a Java application
- Add AWS SDK v2 operations to the Java application
- Trigger the AWS SDK v2 operation at runtime (I tested with both SQS and DDB)
Expected behavior
- AWS SDK Span is created and transmitted to the collector
- HTTP Client Span is created and transmitted to the collector
Actual behavior
- AWS SDK Span is created and transmitted to the collector
- HTTP Client Span is created, it is suppressed
- The Java Agent does trace a span and it is visible in debug logs, but that span is suppressed
- “DEBUG io.opentelemetry.javaagent.shaded.instrumentation.api.internal.SupportabilityMetrics - Suppressed Spans by 'io.opentelemetry.apache-httpclient-4.0'”
- The Java Agent does trace a span and it is visible in debug logs, but that span is suppressed
Javaagent or library instrumentation version
v1.31.0
Environment
JDK: Corretto-17.0.8.8.1
OS: Amazon Linux 2023
Additional context
I believe this is related to the addition of this commit in v1.31 https://github.com/open-telemetry/opentelemetry-java-instrumentation/commit/6f0fd8e63179f123a19388e8a07a4e1a97eb3724
I had been using the HTTP spans generated starting 1.28, so the suppression of the spans without any control caused a loss in visibility. As far as I know, the AWS SDK spans do not map one-to-one with requests made to the backend, so there’s a loss in visibility for retries.
The PR didn’t list any details, so it’s not clear to me why the nested HTTP Span needed to be suppressed - I would like to get some more context on that https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/9634
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing commit 6f0fd8e and PR #9634, then reproduce AWS SDK v2 calls with SQS or DynamoDB using Java agent v1.31. Check the suppression metric for apache-httpclient-4.0 and compare the emitted AWS SDK and HTTP spans. Done means the intended behavior is established and, if appropriate, HTTP spans for retries are emitted and covered by instrumentation tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100