dotnet / dotnet/diagnostics

`!PrintException -lines` drops source-line bracket for unwound stack-trace methods in heap minidumps

Open
#5,909 0 comments 0 reactions 1 assignee Claimed by @max-charlamb View on GitHub
bug
Dominant language
C++
Stars
1.3k
Forks
404
Avg merge
2d 5h
Merged PRs (30d)
35

Description

## Summary

`!PrintException -lines` intermittently fails to resolve the `[file @ N]` source-line bracket for individual frames in an exception's `StackTrace`. The failure is **deterministic for any given dump** but appears flaky in CI because it depends on whether the affected method is currently on a live thread's stack and on the JIT's native-code offset for the exception throw point.

Sample repro below shows the bug fires with **high reliability** when the throwing method is unwound off the live stack and is large enough that its return IP lies in a different NibbleMap byte than its entry IP. This affects all CoreCLR runtimes when consuming heap-type (not heap2) minidumps.

Found after investigating CI failure on `SOSExceptionTests.TaskNestedException(config: projectk.sdk.prebuilt.9.0.14)` (Windows x86 Release leg, internal pipeline).

## Reproducer

### Debuggee (`HeavyExn.csproj`, net9.0/win-x86/Release)

```csharp
using System;
using System.Runtime.CompilerServices;

internal static class Program
{
[MethodImpl(MethodImplOptions.NoInlining)]
static long HeavyMethod(long start)
{
long x = start;
// 400 lines of trivial arithmetic, expanding to ~30 KB of x86 codegen
x = x * 31 + 0; if (x == long.MinValue) Environment.Exit(99);
x = x * 31 + 1; if (x == long.MinValue) Environment.Exit(99);
// ...
x = x * 31 + 399; if (x == long.MinValue) Environment.Exit(99);
throw new InvalidOperationException("thrown from end of HeavyMethod");
}

[MethodImpl(MethodImplOptions.NoInlining)]
static Exception Capture(long seed)
{
try { HeavyMethod(seed); return null; }
catch (Exception e) { return e; }
}

static void Main(string[] args)
{
long seed = args.Length > 0 ? long.Parse(args[0]) : 0;
Exception inner = Capture(seed);
throw new AggregateException("outer holds the inner", inner);
}
}
```

### Capture script

```powershell
$env:DOTNET_DbgEnableMiniDump = '1'
$env:DOTNET_DbgMiniDumpType = '2' # Heap (MiniDumpWithPrivateReadWriteMemory)
$env:DOTNET_DbgMiniDumpName = "C:\dumps\heavy.dmp"
.\HeavyExn.exe
```

### Investigation

```text
0:000> !PrintException -lines
StackTrace (generated):
SP IP Function
02F7E580 08D8A4FB HeavyExn!HeavyExn.Program.HeavyMethod(Int64)+0x76e3 ← NO bracket
02F7F218 08D82DEE HeavyExn!HeavyExn.Program.Capture(Int64)+0x26
[C:\...\Program.cs @ 418] ← bracket OK
```

### Repro rate (Windows x86 Release)

Ran `HeavyExn` 30 times, capturing a heap dump and running `!PrintException -lines` on each via cdb + locally-built SOS:

| Runtime | `DOTNET_EnableFastHeapDumps` | HeavyMethod bracket present | Failure rate |
|---|---|---|---|
| net9.0.17 | unset | 0 / 30 | **100%** |
| net9.0.17 | `1` (env var ignored on 9.0) | 0 / 30 | **100%** |
| net8.0.28 | unset | 0 / 30 | **100%** |
| net8.0.28 | `1` | 30 / 30 | **0%** |

The bug is **deterministic** for the same target binary. CI flakiness comes from method-size proximity to the NibbleMap-granularity boundary — `TaskNestedException`'s `RandomUserTask.InnerException()` at `+0x54` happens to straddle the boundary, while a larger method like `HeavyMethod` at `+0x76e3` fails every time.

## Root cause

On Windows, `createdump` spawns and calls `MiniDumpWriteDump(..., NULL, NULL, NULL)`. The OS `dbgcore.dll` recognizes loaded `coreclr.dll` and invokes the CLR DAC's `ICLRDataEnumMemoryRegions::EnumMemoryRegions` to collect runtime-aware memory regions. On net9 with heap dump type, this goes through `ClrDataAccess::EnumMemoryRegions` → `EnumMemoryRegionsWorkerHeap(CLRDATA_ENUM_MEM_HEAP)` ([`enummem.cpp`](https://github.com/dotnet/runtime/blob/release/9.0/src/coreclr/debug/daccess/enummem.cpp)).

`EnumMemoryRegionsWorkerHeap` then calls `EnumMemDumpAllThreadsStack(flags)`, which for **each managed thread**:

1. Walks the current stack and for **live frames** calls `EECodeInfo(actualReturnIP)` — this touches the NibbleMap byte for the return-IP and pulls it into the dump.
2. Walks each **exception's `_stackTrace`** via `DumpManagedExcepObject` ([source](https://github.com/dotnet/runtime/blob/release/9.0/src/coreclr/debug/daccess/enummem.cpp)). For each `StackTraceElement`, it does:

```cpp
for (size_t i = 0; i < stackTrace.Size(); i++)
{
MethodDesc* pMD = stackTrace[i].pFunc;
if (!DacHasMethodDescBeenEnumerated(pMD) && DacValidateMD(pMD))
{
pMD->EnumMemoryRegions(flags);
FindLoadedMethodRefOrDef(pMD->GetMethodTable()->GetModule(), pMD->GetMemberDef());
DebugInfoManager::EnumMemoryRegionsForMethodDebugInfo(flags, pMD);
PCODE addr = pMD->GetNativeCode(); // ← METHOD ENTRY, not stackTrace[i].ip
if (addr != (PCODE)NULL)
{
EECodeInfo codeInfo(addr); // ← touches NibbleMap for ENTRY only
if (codeInfo.IsValid())
{
IJitManager::MethodRegionInfo methodRegionInfo = { 0 };
codeInfo.GetMethodRegionInfo(&methodRegionInfo);
}
}
}
DacEnumCodeForStackwalk(stackTrace[i].ip); // small window around IP
}
```

The `EECodeInfo` is constructed from `pMD->GetNativeCode()` (the **method entry**), not from `stackTrace[i].ip` (the **actual return IP**). `EECodeInfo::Init` walks the RangeSection and reads the NibbleMap byte covering the entry. For methods larger than `NIBBLE_GRANULARITY` (~16 bytes per byte / 256 bytes per cache line on x86), the byte for `entry` and the byte for `entry+0x54` (or `entry+0x76e3`) live in different addresses — only the entry-IP byte is pulled in.

`DacEnumCodeForStackwalk(stackTrace[i].ip)` does not touch the NibbleMap for the IP; it grabs a small window of executable bytes.

### Why live frames are unaffected

`EnumMemWalkStackHelper` calls `EECodeInfo(addr)` where `addr = GetControlPC(®Disp)` — the **actual return address**. The NibbleMap byte for that exact return IP gets captured.

### Why small methods are unaffected

If method size ≤ NibbleMap granularity, the entry NibbleMap byte covers all return IPs within the method too.

### What `!PrintException -lines` does

Per-frame, it calls `GetLineByOffset(ste.ip)` → `ConvertNativeToIlOffset(ip)` → `GetClrMethodInstance(ip)` → DAC `IXCLRDataProcess::StartEnumMethodInstancesByAddress(ip)` → `ExecutionManager::GetCodeMethodDesc(ip)` → `FindCodeRange(ip)` → `EEJitManager::FindMethodCode` which **reads the NibbleMap byte for `ip`**. If that byte isn't in the dump, this returns `FALSE` → `GetClrMethodInstance` fails → no IL-offset mapping → no `[file @ N]` bracket.

### Confirming the mechanism with cdb

```
0:000> dd 0907E910 L8 # RealCodeHeader for InnerException
0907e910 0907e938 00000000 0907e920 090828d4
0907e920 ???????? ???????? ???????? ???????? ← GCInfo blob, NOT captured

0:000> dd 08A3AB1C L1 # NibbleMap byte for return IP 0x09041A14
08a3ab1c ???????? ← NibbleMap entry, NOT captured

0:000> !IP2MD 090419C0 # method entry
MethodDesc: 090828d4
Source file: ...RandomUserTask.cs @ 37 ← entry-NibbleMap-cell IS captured

0:000> !IP2MD 09041A14 # return IP
Failed to request MethodData, not in JIT code range
```

The DAC has *only* the method entry's NibbleMap byte — not the one for the return IP.

## Fix options

### A. SOS-side fallback (preferred)

Thread an optional `MethodDesc` hint through `ConvertNativeToIlOffset`. When `GetClrMethodInstance(ip)` (the IP→RangeSection lookup) fails and the caller supplied a hint, derive an `IXCLRDataMethodInstance` from the MD instead:

```cpp
HRESULT
ConvertNativeToIlOffset(ULONG64 nativeOffset, BOOL bAdjust,
ULONG64 methodDescHint, // NEW
IXCLRDataModule** ppModule,
mdMethodDef* methodToken, PULONG32 methodOffs)
{
ToRelease pMethodInst(NULL);
HRESULT Status = GetClrMethodInstance(nativeOffset, &pMethodInst);
if (Status != S_OK && methodDescHint != 0)
{
// Bypass the RangeSection walk via MD → Module → MethodDef → EnumInstance
Status = GetClrMethodInstanceFromMethodDesc(methodDescHint, &pMethodInst);
}
if (FAILED(Status)) return Status;
// ... rest unchanged: GetILOffsetsByAddress / GetTokenAndScope / etc.
}
```

`FormatException` passes `(ULONG64)ste.pFunc` as the hint.

**Pros**:
- ~30 LOC in SOS only, no DAC/runtime changes
- Works on every existing dump in the wild (no runtime/createdump update needed)
- Backward compatible (existing callers default the param to 0)
- Preserves IP-keyed path as the primary (it's more precise for tiered methods when memory is available)

**Cons**:
- Resolves `@ 37` instead of `@ 38` in the failing TaskNestedException case (off by one due to sequence-point boundary differences when computing IL offset on the MD-derived instance — needs investigation, or accept it and loosen the test regex)
- Only applies when caller has a MD (i.e. exception stack-trace path); doesn't help generic `GetLineByOffset(ip)` callers

**Validated locally**: 100% → 0% failure on HeavyExn repro; CI dump now resolves `[file @ 37]`.

### B. Runtime-side: extra `EECodeInfo` enumeration in DAC for stack-trace IPs

Modify the DAC's `DumpManagedExcepObject` loop to also call `EECodeInfo(stackTrace[i].ip)` (using the actual return IP, not the entry). This touches the correct NibbleMap byte and pulls it into the dump.

```cpp
for (size_t i = 0; i < stackTrace.Size(); i++)
{
// ... existing ...
if (pMD->GetNativeCode() != (PCODE)NULL)
{
EECodeInfo entryCodeInfo(pMD->GetNativeCode()); // existing
if (entryCodeInfo.IsValid()) entryCodeInfo.GetMethodRegionInfo(...);

// NEW: touch the NibbleMap byte for the actual return IP
EECodeInfo ipCodeInfo(stackTrace[i].ip);
if (ipCodeInfo.IsValid()) ipCodeInfo.GetMethodRegionInfo(...);
}
DacEnumCodeForStackwalk(stackTrace[i].ip);
}
```

**Pros**:
- Fixes the root cause for all consumers, not just `!PrintException -lines`
- Trivial code change

**Cons**:
- Runtime fix → requires backport to release/9.0 (likely won't happen) and won't help dumps already produced
- Doesn't help dumps from older runtimes

### C. Test/runtime opt-in: use HEAP2 via `DOTNET_EnableFastHeapDumps`

Set `DOTNET_EnableFastHeapDumps=1` on the debuggee for net8 / net10 test configurations. `EEJitManager::EnumMemoryRegions` under `CLRDATA_ENUM_MEM_HEAP2` dumps every code heap's entire NibbleMap (`heap->pHdrMap`) wholesale, eliminating the gap.

```diff
- DOTNET_DbgEnableMiniDump=1, DOTNET_DbgMiniDumpType=2
+ DOTNET_DbgEnableMiniDump=1, DOTNET_DbgMiniDumpType=2, DOTNET_EnableFastHeapDumps=1
```

**Pros**:
- Validated: 0% failure on net8 with this env var
- No SOS or DAC change

**Cons**:
- **Doesn't fix net9** — `g_EnableFastHeapDumps` global was added in 8.0 and 10.0 but never backported to 9.0 ([net9 enummem.cpp](https://github.com/dotnet/runtime/blob/19c07820cb72aafc554c3bc8fe3c54010f5123f0/src/coreclr/debug/daccess/enummem.cpp) has no reference to it)
- Doesn't fix dumps already produced by users in the field
- Adds environment-dependent test behavior
- Doesn't help users debugging their own production dumps

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.