[Blazor] Better diagnostics for Blazor server
- Dominant language
- C#
- Stars
- 38.4k
- Forks
- 10.9k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 276
Description
We get a lot of questions/issues opened about Blazor Server memory management. It is clear that this is a confusing aspect for customers and that they run into issues when they inadvertently introduce problematic patterns in their apps, like components rendering a large set of elements which can cause the memory to grow significantly.
When a customer finds itself in this situation, they generally open an issue with us that suggest there is a memory leak in the framework (which has not been the case so far) and we get into a discussion where we explain how Blazor deals with circuits, how memory works in Server GC and how to troubleshoot memory issues. The general points that people miss are:
* Circuits are alive for a given time after the session ends (up to a maximum number of circuits) to support reconnection.
* The server will only collect the circuits when it triggers a collection for the generation they live in, and in general, they will tend to go to Gen2.
* Server GC will only trigger a Gen2 collection when the server is running low on memory.
It will be extremely helpful if we can add additional diagnostics to help bring visibility into some of the internal aspects of how Blazor server operates. Specifically:
* dotnet-counters to track on a given server instance over time:
* The number of connected circuits.
* The number of disconnected circuits.
This makes it trivial to prove that we are handling the circuit lifetime correctly (or to spot when we are not) and since all the allocated framework memory is rooted by a circuit, it makes it easy to prove that the memory issue is not on the framework.
Further diagnostics to help identify problematic circuit instances using clrMD to identify circuits consuming a large amount of memory. We can do this as follows:
* Attach to the live process (or a memory dump).
* Walk the heap to find all the CircuitHost instances.
* For each CircuitHost instance, introspect its renderer and look at the ComponentState instances.
* For each ComponentState, we look at the backing field for the CurrentRenderTree
* On the RenderTreeBuilder we look at RenderTreeFrameArrayBuilder we look at the _items field to check the length of the array as well as the `_itemsInUse`.
* With this information, we can point out when a component rendered a large number of elements in the past or is currently rendering a large number of elements.
* If it is currently rendering a large number of elements, `_itemsInUse` will be large.
* If it has rendered a large number of elements in the past `_items` will be large.
With this data, we should be able to point out what components rendered a large number of elements currently or in the past as well as what circuit they belong(ed) to and its current state (whether the circuit is active, disconnected, or waiting to be collected) and it should also help discard the framework or the component rendering as the source of the memory issues.
Note that this could also be extended to use something similar to `!objsize` on a component instance to track how much memory that component is allocating, which in turn would help identify components in the app that are holding on to a lot of memory (think for example a component that has a large list in memory).
This last bit is not only useful for Blazor Server but it might be useful too for Blazor Webassembly and Desktop.
Another way to identify issues with the app this issue would be to check the `ArrayPool` to identify large buffers that might have been allocated by the app in the past.
Examples of issues in this area in the past (there are likely more):
https://github.com/dotnet/aspnetcore/issues/43221
https://github.com/dotnet/aspnetcore/issues/39238
https://github.com/dotnet/aspnetcore/issues/39156
https://github.com/dotnet/aspnetcore/issues/33547
https://github.com/dotnet/aspnetcore/issues/33077
https://github.com/dotnet/aspnetcore/issues/32588
https://github.com/dotnet/aspnetcore/issues/30210
https://github.com/dotnet/aspnetcore/issues/21991
https://github.com/dotnet/aspnetcore/issues/18556
https://github.com/dotnet/aspnetcore/issues/17945
https://github.com/dotnet/aspnetcore/issues/14545
https://github.com/dotnet/aspnetcore/issues/10623
Contributor guide
Assessment
This issue has not been assessed yet.