open-telemetry / open-telemetry/opentelemetry-java

gRPC library server-side spans created via the opencensus shim not set as parents of spans created in user code

Open
#6,416 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug
Dominant language
Java
Stars
2.5k
Forks
1k
Avg merge
3d 17h
Merged PRs (30d)
58

Description

Describe the bug
The gRPC libraries call opencensus APIs to add spans for incoming server requests (these spans have a remote parent). These have a name like "Recv.MyService.MyMethod'.

We wish to add further child spans in our user code (e.g. our implementation of MyService.MyMethod).

The problem we see is that our child spans are created as brand new traces.

Steps to reproduce
Reproduction requires a working gRPC server (it can just be a hello world) with opencensus instrumentation enabled.

To reproduce the problem:

  1. Add the opencensus-shim as a dependency
  2. Start & activate a span in the implementation of any gRPC service method.

What did you expect to see?

The spans created in the user code should be children of the spans created by the gRPC server interceptors.

What did you see instead?

The spans were treated as root spans and a new trace was started.

What version and what artifacts are you using?

    implementation platform("io.opentelemetry:opentelemetry-bom:1.37.0")
    implementation platform("io.grpc:grpc-bom:1.55.3")

    implementation "io.grpc:grpc-census"
    implementation "io.grpc:grpc-protobuf"
    implementation "io.grpc:grpc-stub"

    implementation "io.opencensus:opencensus-impl:0.31.1"

    implementation "io.opentelemetry:opentelemetry-api"
    implementation "io.opentelemetry:opentelemetry-sdk"
    implementation "io.opentelemetry:opentelemetry-exporter-logging"
    implementation "io.opentelemetry:opentelemetry-exporter-otlp"
    implementation "io.opentelemetry:opentelemetry-opencensus-shim:1.37.0-alpha"

    compileOnly "javax.annotation:javax.annotation-api:1.3.2"

    runtimeOnly "io.grpc:grpc-netty-shaded"

(from build.gradle)

Environment

Java 21 (temurin)

Additional context
Please see also: https://github.com/grpc/grpc-java/issues/7732 and #5475

It does seem that the bug is really in the gRPC library since it bypasses the context management API when it activates the spans. Perhaps there is something we can do to make the fix easier for them (I think gRPC's context is passed around a lot so switching that to another type may be quite painful).

The client side does NOT have the same issue, but it seems like this may be good luck rather than intention. On the client side we create a span using otel API then call into the grpc client library. These spans created by the client library are correctly added as children but this happens despite the fact that gRPC itself thinks there is no parent (it calls the span builder with a blank parent span). It ends up with the correct output because the shim then finds the original parent via the opentelemetry context.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided hello-world gRPC server reproduction using the OpenTelemetry OpenCensus shim and the listed gRPC dependencies. Trace how the server interceptor activates its span and how user code starts a span; done means user spans inherit the incoming server span rather than starting new traces.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.