dapr / dapr/go-sdk

Traceparent not forwarded via PubSub

Open
#355 5 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Go
Stars
479
Forks
187
PR merge metrics
No merged PRs in 30d

Description

I don't know if this is a *bug* or *feature request*. I have two services "server" and "echo". Server sends send a message via pubsub to echo. Both services are fully instrumented with OpenTelemetry. A call to server give me the following trace

![image](https://user-images.githubusercontent.com/1181349/215324386-2af1f5c2-8067-4e43-8da3-a361c6b3f9cd.png)

The last method SendNotifcation forwards the instrumented ctx to the DAPR client:

```
func (n *Sender) SendNotification(ctx context.Context, message []byte) error {
ctx, span := otel.Tracer("sender").Start(ctx, "SendNotification")
defer span.End()

llog := log.WithFields(log.Fields{
"pubsubName": n.PubSubName,
"topicName": n.TopicName,
"message": string(message),
})
llog.Debug("Sending sender")
err := n.client.PublishEvent(ctx, n.PubSubName, n.TopicName, message)
if err != nil {
llog.WithError(err).Warn("Unable to send sender")
return err
}
return nil
}
```

But the transported message has a new traceid:

```
{
"data": "{\"Before\":{\"Id\":\"77\",\"Title\":\"Title\",\"Description\":\"The description\",\"Status\":\"\"},\"After\":{\"Id\":\"77\",\"Title\":\"Title\",\"Description\":\"The description\",\"Status\":\"\"},\"ChangeType\":\"CREATE\"}",
"datacontenttype": "text/plain",
"id": "b946b14e-1770-4316-8483-9eca55602318",
"pubsubname": "todo-pubsub",
"source": "todo",
"specversion": "1.0",
"time": "2023-01-29T11:22:03Z",
"topic": "todo",
"traceid": "00-f4905fb842564c59ca0905e18f408be7-24f05a94522b81d3-01",
"traceparent": "00-f4905fb842564c59ca0905e18f408be7-24f05a94522b81d3-01",
"tracestate": "",
"type": "com.dapr.event.sent"
}
```

Receiving the trace works again:

![image](https://user-images.githubusercontent.com/1181349/215324484-06a6c0f0-a47f-4b46-a311-b7c54fcc95b1.png)

But alltogether I have two traces:

![image](https://user-images.githubusercontent.com/1181349/215324508-fd33d191-cf73-468b-94e1-4ebf9f7c918f.png)

Questions:

* Is this feature missing?
* Am I doing something wrong?
* Does this make sense at all having a trace over asynchronous communication?

Contributor guide

Open the contributing guide

Research direction

Start with Sender.SendNotification and the n.client.PublishEvent call shown in the report, then trace how the Go SDK handles context propagation for PubSub. Reproduce the two-trace case with the reported server/echo setup and verify whether the outgoing traceparent is preserved across the asynchronous message. Done means the behavior is clarified or corrected and covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.