microsoft / microsoft/durabletask-dotnet

Performance: avoid repeated orchestration-history scans for tracing

Open
#769 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Needs: Triage :mag:
Dominant language
C#
Stars
193
Forks
60
Avg merge
3d 12h
Merged PRs (30d)
4

Description

1. What is the issue?

The worker scans the full orchestration history to locate scheduling events for every relevant new completion event, even when no OpenTelemetry listener is active.

2. Likely priority

High for long-running orchestrations with large histories and fan-in batches.

3. Impact on performance

For a work item with N new completion events and M past events, tracing work can become O(N x M). The scans and LINQ iterator allocations occur before ActivitySource.StartActivity determines that tracing is disabled, so the cost is paid in the normal no-listener case.

4. Details and code reference

Relevant worker path:

https://github.com/microsoft/durabletask-dotnet/blob/883211a3a7ab0b55bb90b9c9e498148e4d65ead9/src/Worker/Grpc/GrpcDurableTaskWorker.Processor.cs#L589-L689

OnRunOrchestratorAsync scans PastEvents from local lookup functions invoked inside the NewEvents loop. It also scans NewEvents.Concat(PastEvents) to find ExecutionStarted before starting the tracing activity.

Relevant tracing helper:

https://github.com/microsoft/durabletask-dotnet/blob/883211a3a7ab0b55bb90b9c9e498148e4d65ead9/src/Shared/Grpc/Tracing/TraceHelper.cs#L67-L114

The tracing helper only discovers whether an activity can be created after these lookups.

5. Guidance for how to fix

Expose a cheap tracing-listener check from TraceHelper and skip all trace-event lookup work when no listener is registered. When tracing is enabled, build indexes once per work item for task-scheduled and sub-orchestration-created events, preserving the current first/last lookup semantics, rather than rescanning history for each new event. Add coverage for duplicate IDs, ordering semantics, and no-listener behavior.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/Worker/Grpc/GrpcDurableTaskWorker.Processor.cs, especially OnRunOrchestratorAsync, and read src/Shared/Grpc/Tracing/TraceHelper.cs to understand when tracing listeners are detected. Add coverage for duplicate IDs, ordering semantics, and the no-listener path. Done means avoiding trace lookups without listeners and indexing relevant events once when tracing is enabled while preserving current lookup behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, distributed-systems, observability, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.