Azure / Azure/durabletask

Avoid serializing inline Azure Storage queue messages twice

Open
#1,378 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
2d 23h
Merged PRs (30d)
6

Description

## What is the issue

`MessageManager.SerializeMessageDataAsync` serializes every `MessageData` instance into `rawContent`, uses that string to calculate its UTF-8 size and select inline versus blob storage, and then serializes the same object a second time in the common `InlineJson` branch:

https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.AzureStorage/MessageManager.cs#L103-L123

The only mutation between those two serializations is `TotalMessageSizeBytes`. That property is internal and is not a `[DataMember]`, while `MessageData` is a `[DataContract]` whose wire properties are explicitly marked:

https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.AzureStorage/MessageData.cs#L24-L27

https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.AzureStorage/MessageData.cs#L94-L108

Each serialization also creates a fresh `StringBuilder`, `StringWriter`, and result string:

https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.AzureStorage/Utils.cs#L168-L184

This method is called for every outbound Azure Storage task-hub queue message:

https://github.com/Azure/durabletask/blob/5217032961abf45846f462732a5e2813316e3747/src/DurableTask.AzureStorage/Messaging/TaskHubQueue.cs#L91-L110

The blob-offload branch legitimately serializes a small wrapper after uploading `rawContent`; only the inline branch repeats the original serialization.

## Performance impact

Inline messages are the normal path for payloads below the 45 KiB threshold. Every such orchestration, activity, timer, sub-orchestration, and external-event message currently incurs two complete Newtonsoft JSON traversals and two sets of temporary buffers/strings instead of one.

The duplicate CPU and allocation cost scales linearly with message throughput and payload size. At high task-hub throughput this increases serialization CPU, memory bandwidth, Gen-0 pressure, and queue-send latency without changing the resulting wire payload.

## Proposed backward-compatible solution

Return the already-produced `rawContent` in the `InlineJson` branch:

```csharp
if (messageFormat != MessageFormatFlags.InlineJson)
{
// Existing blob upload and wrapper serialization remain unchanged.
}

return rawContent;
```

This preserves the exact JSON that was already used for the byte-count/format decision. It changes no public API, queue schema, serializer settings, or blob behavior.

## Validation

- Add regression coverage for both `UseDataContractSerialization` modes and representative custom type-binder settings, asserting that the optimized inline result exactly matches the current second serialization.
- Cover payloads immediately below and above the inline threshold to ensure the blob-wrapper path is unchanged.
- Benchmark representative small, medium, and near-threshold messages with allocation diagnostics; the inline path should perform one `MessageData` serialization instead of two.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/DurableTask.AzureStorage/MessageManager.cs at SerializeMessageDataAsync, then inspect MessageData.cs, Utils.cs, and Messaging/TaskHubQueue.cs to understand the inline and blob paths. Add regression coverage for both serialization modes, binder settings, and threshold boundaries, then compare the inline output and run allocation benchmarks. Done means inline messages serialize once while blob behavior and wire output remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, cloud, performance
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.