apache / apache/arrow-dotnet

NativeMemoryAllocator doesn't remove all memory pressure

Open
#50 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
39
Forks
30
Avg merge
1d 9h
Merged PRs (30d)
16

Description

### Describe the bug, including details regarding any error messages, version, and platform.

The `NativeMemoryAllocator` always over-allocates by the alignment amount to ensure the final buffer is aligned. When it [adds GC pressure](https://github.com/apache/arrow-dotnet/blob/710f6e47bd20c5a0dca471a27d67c2fff65d581c/src/Apache.Arrow/Memory/NativeMemoryAllocator.cs#L45) via `GC.AddMemoryPressure()`, this extra padding is accounted for. However, when `NativeAllocationOwner` [releases the memory allocation](https://github.com/apache/arrow-dotnet/blob/710f6e47bd20c5a0dca471a27d67c2fff65d581c/src/Apache.Arrow/Memory/NativeMemoryAllocator.cs#L58), it doesn't include the extra padding when it calls `GC.RemoveMemoryPressure()` (the `length` parameter is the size of the `Memory`, not the unmanaged allocation).

This doesn't appear to have long-term consequences, since the GC will periodically clear its unmanaged memory pressure counters, but my understanding is it can lead to extra garbage collections overall.

A tool like [`PerfView`](https://github.com/microsoft/perfview) can show events for each call to `AddMemoryPressure` and `RemoveMemoryPressure`. Running it with the following code shows a call to add memory pressure of 74 bytes and a call to remove memory pressure of 10 bytes:

```csharp
using Apache.Arrow.Memory;

MemoryAllocator.Default.Value.Allocate(10).Dispose();
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.