microsoft / microsoft/durabletask-dotnet

Entity operations don't propagate trace context from parent orchestration

Open
#653 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Needs: Triage :mag:
Dominant language
C#
Stars
193
Forks
60
Avg merge
3d 12h
Merged PRs (30d)
4

Description

Summary

When an orchestration calls an entity via CallEntityAsync(), the DTS SDK does not propagate the orchestration's trace context (ParentTraceContext) to the entity operation. This causes entity spans (and any child spans created within the entity) to appear as disconnected root traces instead of being parented under the calling orchestration's trace tree.

Root Cause

In ProtoUtils.ConstructOrchestratorResponse, ParentTraceContext = CreateTraceContext() is set for:

  • ScheduleTaskAction (activities)
  • CreateSubOrchestrationAction (sub-orchestrations)
  • SendEntityMessageActionno trace context is set

This means the W3C traceparent / tracestate that enables distributed trace stitching is never included in entity messages dispatched from orchestrations.

Expected Behavior

Entity operations dispatched from an orchestration should inherit the orchestration's trace context, just like activities and sub-orchestrations do. The invoke_agent spans (or any other work done inside the entity) should appear as children of the orchestration span in a trace viewer.

Actual Behavior

Entity operations start a new, independent trace. In a trace viewer (e.g., Aspire Dashboard, Jaeger), entity spans appear as separate root traces with no parent relationship to the orchestration that dispatched them.

Impact

This is particularly impactful for scenarios that use Durable Entities as the execution model for agent invocations (e.g., DurableAIAgent in the Microsoft Agent Framework). Each invoke_agent span appears disconnected from the orchestration trace, making it impossible to view the full execution tree in a single trace.

Suggested Fix

In ProtoUtils.cs, add ParentTraceContext = CreateTraceContext() to the SendEntityMessage action construction (similar to how it's done for ScheduleTask and CreateSubOrchestration). The entity work item processing path would also need to parse and restore this context when executing entity operations.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/Shared/Grpc/ProtoUtils.cs and compare SendEntityMessageAction with ScheduleTaskAction and CreateSubOrchestrationAction, where ParentTraceContext is already created. Then trace the entity work item processing path to see how the context is parsed and restored. Done means entity spans and child spans are parented under the calling orchestration trace.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, grpc
Domain
distributed-systems, observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.