Heavy LOH allocations in MessageCenter.TryForwardRequest
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 354
Description
I'm currently analyzing a production workload that trips over from LOH allocations at a certain point. This seems to correlate heavily with the number of grains in an individual silo, ~10,000 being the "magic number". The deployment has grown to 80 silo instances hosting >500,000 grains.
The extracted memory allocation profiling pretty much entirely points to that one method as the sole cause for everything:



It looks before inlining that it's actually [`Message.AddToCacheInvalidationHeader`](https://github.com/dotnet/orleans/blob/v9.1.2/src/Orleans.Core/Messaging/Message.cs#L277), which seems to employ a pretty bad `List` pattern:
- new List (size 0, capacity 4)
- AddRange (size according to argument, capacity == size)
- Add (size + 1, capacity *= 2)
which allocates thrice the needed amount and wastes half of the list's capacity. It could create the new list with the right capacity already.
Is this indicative of a different issue in our setup? Is there a way to avoid / reduce the allocations beyond just setting the right capacity from the get go?
Contributor guide
Research direction
Start in src/Orleans.Core/Messaging/Message.cs at Message.AddToCacheInvalidationHeader, using the reported allocation pattern as the initial investigation target. Check whether the list capacity can match the incoming range and subsequent item, then verify that the change reduces LOH allocations for the described grain counts without changing cache invalidation behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100