Remove per-message Task.Run from Service Bus inline message deserialization
- Dominant language
- C#
- Stars
- 1.7k
- Forks
- 335
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 6
Description
## What is the issue
`ServiceBusUtils.LoadMessageStreamAsync` queues synchronous, in-memory work to the thread pool for every message whose body is stored inline:
https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.ServiceBus/Common/ServiceBusUtils.cs#L290-L307
For `netstandard2.0`, the work is only `new MemoryStream(message.Body)`. For `net48`, `message.GetBody()` reads the already-received brokered-message body. Neither branch performs asynchronous I/O, but both use `Task.Run`.
The orchestration and tracking receivers deserialize whole batches through this method using `Task.WhenAll`:
https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs#L543-L561
https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs#L1358-L1372
The configured prefetch count is 50, so a full batch can enqueue 50 trivial thread-pool work items at once:
https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.ServiceBus/Settings/ServiceBusOrchestrationServiceSettings.cs#L70-L73
## Performance impact
Each inline message creates and schedules an unnecessary work item plus its task/delegate state. Under sustained load, batches from multiple dispatchers create bursts of thread-pool queueing that add scheduling latency, consume worker threads, and increase short-lived allocations. Thread-pool ramp-up or contention can amplify first-batch and tail latency even though there is no I/O to overlap.
The overhead scales with message rate and is paid before every inline task-message deserialization. The external-blob path is genuinely asynchronous and is not affected by this concern.
## Proposed backward-compatible solution
Keep the existing private `Task` signature and return an already-completed task for inline bodies:
```csharp
#if NETSTANDARD2_0
return Task.FromResult(new MemoryStream(message.Body));
#else
return Task.FromResult(message.GetBody());
#endif
```
This preserves the same stream construction and downstream deserialization behavior while removing the thread-pool hop. Leave the blob-store load path unchanged.
## Validation
- Add coverage for both target-framework branches confirming that inline bodies deserialize identically and the returned task is already complete.
- Retain integration coverage for external blob-backed messages.
- Benchmark batch deserialization at the default prefetch size, comparing elapsed time, allocations, thread-pool work-item count, and tail latency.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.