dotnet / dotnet/orleans

Heavy LOH allocations in MessageCenter.TryForwardRequest

Closed
#9,569 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

I'm currently analyzing a production workload that trips over from LOH allocations at a certain point. This seems to correlate heavily with the number of grains in an individual silo, ~10,000 being the "magic number". The deployment has grown to 80 silo instances hosting >500,000 grains.

The extracted memory allocation profiling pretty much entirely points to that one method as the sole cause for everything:

![Image](https://github.com/user-attachments/assets/d4cfab87-2448-4aa1-b20d-f734f01ebaf2)
![Image](https://github.com/user-attachments/assets/39c19b8d-adcc-470e-a3d3-7a9e83a43494)
![Image](https://github.com/user-attachments/assets/01da3c87-1fcc-4377-9a52-85edacd8d8a9)

It looks before inlining that it's actually [`Message.AddToCacheInvalidationHeader`](https://github.com/dotnet/orleans/blob/v9.1.2/src/Orleans.Core/Messaging/Message.cs#L277), which seems to employ a pretty bad `List` pattern:
- new List (size 0, capacity 4)
- AddRange (size according to argument, capacity == size)
- Add (size + 1, capacity *= 2)

which allocates thrice the needed amount and wastes half of the list's capacity. It could create the new list with the right capacity already.

Is this indicative of a different issue in our setup? Is there a way to avoid / reduce the allocations beyond just setting the right capacity from the get go?

Contributor guide

Open the contributing guide

Research direction

Start in src/Orleans.Core/Messaging/Message.cs at Message.AddToCacheInvalidationHeader, using the reported allocation pattern as the initial investigation target. Check whether the list capacity can match the incoming range and subsequent item, then verify that the change reduces LOH allocations for the described grain counts without changing cache invalidation behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.