Opentelemetry Trace graph becomes disconnected after creating Grpc Call
- Dominant language
- C#
- Stars
- 4.5k
- Forks
- 836
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 7
Description
### What version of gRPC and what language are you using?
Grpc.AspNetCore 2.65.0
### What operating system (Linux, Windows,...) and version?
Windows 10 22H2
### What runtime / compiler are you using (e.g. .NET Core SDK version `dotnet --info`)
8.0.400
### What did you do?
I instrumented an application with open telemetry for tracing. The tracing tree is disjointed because one activity is being created and started regardless if there are listeners.
### What did you expect to see?
A trace span not to be collected (even started) if there are no listeners.
### What did you see instead?
The graph of code should show the following:
1. A request came to the service
2. Service made a request to another service (for simplicity it was to itself)
3. Service did the same as step 1, but on different endpoint.

In the image you can see 2 root spans. They are not unrelated, and should be part of the tree. This is the instrumentation code:
```cs
builder.Services.AddOpenTelemetry()
.ConfigureResource(o => o.AddService("my_service").Build())
.WithTracing(o => o
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddConsoleExporter()
.AddOtlpExporter(o => o.Endpoint = new Uri("http://localhost:4317")));
```
As a debugging step I tried instrumenting everything, added `.AddSource("*")`. This revealed the culprit:

The graph now is now intact, however it has additional instrumentation that is not really needed.
After digging around, I suspect this is the culprit code: https://github.com/grpc/grpc-dotnet/blob/e9cc7e15796d39f1d2656178f56a45c09147d0fe/src/Grpc.Net.Client/Internal/GrpcCall.cs#L826
To my understanding of OpenTelemetry, if there are no listeners to the source, then it is not created, and because of that the trace tree is intact, but here since we force create and start an activity, the parent ids get messed up, and so does the trace tree.
### Anything else we should know about your project / environment?
Repro code: https://github.com/GedasFX/opentelemetry-dotnet-grpc-failure
Contributor guide
Research direction
Start at src/Grpc.Net.Client/Internal/GrpcCall.cs around line 826 and reproduce the disconnected trace with the linked opentelemetry-dotnet-grpc-failure project. Compare tracing with and without AddSource("*"); done means the gRPC call does not disconnect the parent trace when no listener is registered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- api, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100