dotnet / dotnet/runtime

JIT: (bug) can pass an implicit-byref parameter as a struct return buffer, violating the retbuf stack invariant

Open
#133,961 7 comments 0 reactions 0 assignees View on GitHub
area-CodeGen-coreclr
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

For non-async struct-returning calls, the importer can allow the address of an implicit-byref parameter to be used as the hidden return buffer. Morph then turns that local address into the incoming byref, so the callee may store GC references through a caller-supplied byref while assuming the retbuf points to stack memory.

### Minimal Repro

```csharp
using System;
using System.Runtime.CompilerServices;

public struct Big
{
public object A, B, C, D;
}

public static class Program
{
[MethodImpl(MethodImplOptions.NoInlining)]
public static Big GetBig(object o)
{
Big b;
b.A = o;
b.B = o;
b.C = o;
b.D = o;
return b;
}

[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.AggressiveOptimization)]
public static object Test(Big b, object o)
{
b = GetBig(o);
return b.A;
}

public static void Main() => Console.WriteLine(Test(default, "hello"));
}
```

### Expected

The JIT should reject an implicit-byref local as a return buffer and use a stack temp, then copy the result into `b` with the required write barriers.

### Actual

The caller passes the incoming implicit byref as the retbuf, and the callee writes object references to it with plain stores:

```
mov rcx, rbx ; retbuf == incoming implicit byref
call [Program:GetBig(System.Object):Big]

mov gword ptr [rcx], rdx
mov gword ptr [rcx+0x08], rdx
mov gword ptr [rcx+0x10], rdx
mov gword ptr [rcx+0x18], rdx
```

The simple JIT-to-JIT repro prints `hello`, so the missed-barrier risk is latent unless the implicit byref points at GC heap memory.

### Notes

The async path in `impIsLegalRetBuf` already rejects `GT_LCL_ADDR` of an implicit-byref local, but the non-async path relies on `fgAddrCouldBeHeap`, which returns false for `GT_LCL_ADDR`.
The implicit-byref rejection should apply to non-async retbufs too, or `fgAddrCouldBeHeap` should treat these local addresses as potentially heap-backed.

Contributor guide

Open the contributing guide

Research direction

Start by tracing impIsLegalRetBuf and fgAddrCouldBeHeap for non-async GT_LCL_ADDR values, then run the minimal C# reproducer and inspect the generated code. Done means an implicit-byref local is not used as the retbuf, the result is copied with the required write barriers, and the regression is covered by an appropriate JIT test.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.