open-telemetry / open-telemetry/opentelemetry-java-instrumentation

HTTP Spans no longer emitted by AWS SDK v2 instrumentation

Open
#9,827 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs triage
Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 18h
Merged PRs (30d)
228

Description

Describe the bug

Before v1.31*, calls using the AWS SDK would emit an HTTP Client Span. This span is now suppressed.

*I believe this behavior had existed since v1.28, based on my reading of this PR https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/8405/files#diff-ecc2a9e48ea1cac6e394aabfd4ece49b63cef5f08e950346e5e74cffd3c101e9

Steps to reproduce
  1. Set up the auto-instrumentation on a Java application
  2. Add AWS SDK v2 operations to the Java application
  3. Trigger the AWS SDK v2 operation at runtime (I tested with both SQS and DDB)
Expected behavior
  • AWS SDK Span is created and transmitted to the collector
  • HTTP Client Span is created and transmitted to the collector
Actual behavior
  • AWS SDK Span is created and transmitted to the collector
  • HTTP Client Span is created, it is suppressed
    • The Java Agent does trace a span and it is visible in debug logs, but that span is suppressed
      • “DEBUG io.opentelemetry.javaagent.shaded.instrumentation.api.internal.SupportabilityMetrics - Suppressed Spans by 'io.opentelemetry.apache-httpclient-4.0'”
Javaagent or library instrumentation version

v1.31.0

Environment

JDK: Corretto-17.0.8.8.1
OS: Amazon Linux 2023

Additional context

I believe this is related to the addition of this commit in v1.31 https://github.com/open-telemetry/opentelemetry-java-instrumentation/commit/6f0fd8e63179f123a19388e8a07a4e1a97eb3724

I had been using the HTTP spans generated starting 1.28, so the suppression of the spans without any control caused a loss in visibility. As far as I know, the AWS SDK spans do not map one-to-one with requests made to the backend, so there’s a loss in visibility for retries.

The PR didn’t list any details, so it’s not clear to me why the nested HTTP Span needed to be suppressed - I would like to get some more context on that https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/9634

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing commit 6f0fd8e and PR #9634, then reproduce AWS SDK v2 calls with SQS or DynamoDB using Java agent v1.31. Check the suppression metric for apache-httpclient-4.0 and compare the emitted AWS SDK and HTTP spans. Done means the intended behavior is established and, if appropriate, HTTP spans for retries are emitted and covered by instrumentation tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.