Azure / Azure/durabletask

Remove per-message Task.Run from Service Bus inline message deserialization

Open
#1,376 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
2d 23h
Merged PRs (30d)
6

Description

## What is the issue

`ServiceBusUtils.LoadMessageStreamAsync` queues synchronous, in-memory work to the thread pool for every message whose body is stored inline:

https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.ServiceBus/Common/ServiceBusUtils.cs#L290-L307

For `netstandard2.0`, the work is only `new MemoryStream(message.Body)`. For `net48`, `message.GetBody()` reads the already-received brokered-message body. Neither branch performs asynchronous I/O, but both use `Task.Run`.

The orchestration and tracking receivers deserialize whole batches through this method using `Task.WhenAll`:

https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs#L543-L561

https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs#L1358-L1372

The configured prefetch count is 50, so a full batch can enqueue 50 trivial thread-pool work items at once:

https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.ServiceBus/Settings/ServiceBusOrchestrationServiceSettings.cs#L70-L73

## Performance impact

Each inline message creates and schedules an unnecessary work item plus its task/delegate state. Under sustained load, batches from multiple dispatchers create bursts of thread-pool queueing that add scheduling latency, consume worker threads, and increase short-lived allocations. Thread-pool ramp-up or contention can amplify first-batch and tail latency even though there is no I/O to overlap.

The overhead scales with message rate and is paid before every inline task-message deserialization. The external-blob path is genuinely asynchronous and is not affected by this concern.

## Proposed backward-compatible solution

Keep the existing private `Task` signature and return an already-completed task for inline bodies:

```csharp
#if NETSTANDARD2_0
return Task.FromResult(new MemoryStream(message.Body));
#else
return Task.FromResult(message.GetBody());
#endif
```

This preserves the same stream construction and downstream deserialization behavior while removing the thread-pool hop. Leave the blob-store load path unchanged.

## Validation

- Add coverage for both target-framework branches confirming that inline bodies deserialize identically and the returned task is already complete.
- Retain integration coverage for external blob-backed messages.
- Benchmark batch deserialization at the default prefetch size, comparing elapsed time, allocations, thread-pool work-item count, and tail latency.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.