BufferedStream LOH allocations
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- csharp
- Domain
- performance
Research direction
Start with System.Private.CoreLib/src/System/IO/BufferedStream.cs, especially the source comments about SOH and LOH allocation. Review how BufferedStream creates, uses, and releases its byte[] backing buffer, then inspect related tests in the repository before deciding how large buffers could use ArrayPool.Shared. Done should preserve existing buffering behavior while avoiding the reported repeated LOH allocations; the issue provides measurements and configuration for validation.
Written by the indexing model from the issue text.
Description
Description
Using some measurements, BrotliStream seems to more efficiently when the written data is in larger chunks. This is also described in issue https://github.com/dotnet/runtime/issues/36245
Hence, it seems reasonable to combined writing with BufferedStream, that can buffer data in chunks to the brotli compression.
Based on measurements with a certain type of data, Brotli stream was the most efficient with writes of ~300KB chunks.
However, when such value is set as the buffer size in BufferedStream ctor, the buffer is allocated on the LOH. This results that after a dozen or so compressed streams written (each with a new BufferedStream), a Gen2 GC compaction is triggered. On a hot path, eventually causing performance issues as the GC spends a lot of time collecting garbage.
Configuration
.NET 9, x64, JIT
Regression?
No
Analysis
It seems each new BufferedStream is allocating a byte[] as a backing buffer. Ideally this could be backed by ArrayPool<byte>.Shared instead of new allocations - at least for the case when the buffer is allocated on the LOH. It seems BufferedStream already has knowledge about SOH and LOH as noted in some source code comments.
Workaround
- Pooling
BufferedStreamobjects. - Creating a custom stream object that buffers backed by ArrayPool. However, having a general implementation seems to be relatively complicated.
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- PR merge metrics
- PR metrics pending
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from dotnet/runtime
-
agentic-workflows untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
area-System.Reflection blocking-clean-ci-optional Known Build Error os-mac-os-x untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
area-CodeGen-coreclr untriaged
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
agentic-workflows untriaged
Difficulty 1/5 Under an hour Newbie friendliness 78/100
-
area-VM-meta-mono untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
:watch: Not Triaged 11.0 fundamentals/subsvc
Difficulty 2/5 1-3 hours Newbie friendliness 92/100
dotnet/AspNetCore.Docs#37699 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
SubtitleEdit/subtitleedit#15108 · 1 comment ·
-
area/docs-content Bug pulumi/docs
Difficulty 1/5 1-3 hours Newbie friendliness 94/100
-
Create parent directories only after the containment check in InstallHelper.TryExtractToDirectory Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
PowerShell/PSResourceGet#2056 ·