dotnet / dotnet/runtime

AsAnyMarshaler LayoutImplementation.ConvertToNative leaks native memory on exception

Open
#126,908 2 comments 1 reaction 2 assignees Claimed by @jkoritzinsky View on GitHub
area-Interop-coreclr bug
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

## Description

In `AsAnyMarshaler.LayoutImplementation.ConvertToNative` (`src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs`), native memory allocated via `Marshal.AllocCoTaskMem` is not freed if `LayoutTypeConvertToUnmanaged` throws an exception.

https://github.com/dotnet/runtime/blob/main/src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs#L1032-L1040

The current code:
```csharp
int allocSize = Marshal.SizeOfHelper((RuntimeType)_layoutType, false);
IntPtr pNative = Marshal.AllocCoTaskMem(allocSize);

if (IsIn(dwFlags))
{
StubHelpers.LayoutTypeConvertToUnmanaged(managed, (byte*)pNative, ref _cleanupWorkList);
}

return pNative;
```

If `LayoutTypeConvertToUnmanaged` throws, `pNative` is leaked. The fix should wrap the conversion in a try/catch that frees `pNative` on failure, similar to the pattern used by `ArrayImplementation.ConvertToNative` in the same file:

```csharp
IntPtr pNative = Marshal.AllocCoTaskMem(allocSize);
try
{
if (IsIn(dwFlags))
StubHelpers.LayoutTypeConvertToUnmanaged(managed, (byte*)pNative, ref _cleanupWorkList);
}
catch
{
Marshal.FreeCoTaskMem(pNative);
throw;
}
return pNative;
```

> [!NOTE]
> This issue was filed based on a code review finding by GitHub Copilot. This is a pre-existing issue in the `AsAnyMarshaler` code, not a regression from recent changes.

## Area

area-Interop-coreclr

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.