open-telemetry / open-telemetry/opentelemetry-java-instrumentation

Bug: gRPC Context incorrectly propagated to async threads via OpenTelemetry Context (with reproduction code)

Open
#15,596 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 18h
Merged PRs (30d)
228

Description

When using OpenTelemetry's gRPC instrumentation library with Context.current().wrap(runnable) to propagate OpenTelemetry context to async threads, the gRPC context is incorrectly propagated along with it. This causes errors when making gRPC calls from async threads because the gRPC context from the original request thread is incorrectly activated in the async thread.

To Reproduce:

##reproduction code repository : https://github.com/153952402/grpc-otel-context-bug-reproduction
cd grpc-otel-context-bug-reproduction
mvn spring-boot:run

In another terminal:

grpcurl -plaintext \
  -import-path ./src/main/resources/proto \
  -proto downstream_service.proto \
  -d '{"id":"test-123","data":"test data"}' \
  127.0.0.1:9090 \
  com.example.reproduction.downstream.grpc.DownstreamService/Handle

What did you expect to see?

  1. The gRPC Context should NOT be propagated to async threads - only OpenTelemetry tracing context should be propagated.
  2. When checking io.grpc.Context.current() in async threads, it should return Context.ROOT.
  3. Async threads should be able to make gRPC calls independently without errors.

What did you see instead?

  1. The gRPC Context IS incorrectly propagated to async threads via OpenTelemetry Context.
  2. When checking io.grpc.Context.current() in async threads, it returns a non-ROOT context (the context from the original request thread).
  3. When making gRPC calls from async threads, errors occur because:
    • The gRPC context from the original request is still active
    • This can cause deadline/cancellation states to be incorrectly propagated
    • The context state may be inconsistent for the new gRPC call

What version and what artifacts are you using?

Artifacts:

  • opentelemetry-api version 1.32.0
  • opentelemetry-sdk version 1.32.0
  • opentelemetry-grpc-1.6 (instrumentation library) version 2.1.0-alpha
  • grpc-netty-shaded version 1.60.1
  • grpc-protobuf version 1.60.1
  • grpc-stub version 1.60.1
  • spring-boot-starter-web version 3.2.0
  • grpc-server-spring-boot-starter version 2.15.0.RELEASE

Version:

  • Java: 21
  • Spring Boot: 3.2.0
  • OpenTelemetry: 1.32.0
  • OpenTelemetry Instrumentation (gRPC 1.6): 2.1.0-alpha
  • gRPC: 1.60.1

Thank you for your attention to this issue

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the reproduction repository linked in the issue with the provided Maven and grpcurl commands, then inspect the gRPC instrumentation and OpenTelemetry context-propagation paths in this repository. Done means async threads retain the OpenTelemetry tracing context without inheriting the original gRPC context, report Context.ROOT, and can make independent gRPC calls without errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
distributed-systems, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.