Access violation in WRITE_BARRIER_BIT_REGIONS64 reading the region-to-generation map (uncommitted page)
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- PR merge metrics
- PR metrics pending
Description
### Description
# Access violation in WRITE_BARRIER_BIT_REGIONS64 reading the region-to-generation map (uncommitted page) — .NET 8.0.22, Workstation GC + regions, x64
### Description
Generated by Claude Opus 5 after a few hours of debugging the memory dump:
A long-running Windows service fail-fasted with no managed exception. Analysis of a full WER dump shows the process died from an access violation **inside the JIT write barrier** (`WRITE_BARRIER_BIT_REGIONS64`) while reading the **region-to-generation map**. The byte being read lies on a page that is `MEM_RESERVE` — never committed.
Because an AV inside a write barrier is not recoverable, `ProcessCLRException` routed it to `EEPolicy::HandleFatalError`, so the process was fail-fasted and no managed exception object was ever produced.
@kkokosa This looks like the **regions/x64 counterpart of #123490** (fixed by #131310 for Workstation-segments/x86): a GC bookkeeping-table grow leaving a sub-table's pages uncommitted, with a later access faulting on reserved memory.
### Configuration
| | |
|---|---|
| Runtime | **.NET 8.0.22** — `coreclr.dll` 8.0.2225.52707, `C:\Program Files\dotnet\shared\Microsoft.NETCore.App\8.0.22` |
| OS | Windows 11 Enterprise 10.0.26100.1 (`ge_release`), x64 |
| GC | **Workstation, regions** (`Number of GC Heaps: 1`; 4 MB regions) |
| Barrier variant | `WRITE_BARRIER_BIT_REGIONS64` (bit-marking tail, no software write watch) |
| Host | Windows Service (`ServiceBase.Run`), self-contained-ish deployment on shared framework |
| Uptime before crash | ~1,313,170 s ≈ **15.2 days** |
| Dump | WER full dump (`MiniDumpWithFullMemory` + `MiniDumpWithFullMemoryInfo`) |
Memory state at the crash — no pressure of any kind:
```
GC Allocated Heap Size: 0x56a38bf8 (1,453,558,776 bytes = 1.45 GB)
GC Committed Heap Size: 0x58a99000 (1,487,507,456 bytes = 1.49 GB)
!address -summary
MEM_COMMIT 2691 regions 0`75891000 ( 1.836 GB)
MEM_RESERVE 692 regions 241`38d9a000 ( 2.255 TB)
MEM_FREE 266 regions 7dbe`519c5000 (125.743 TB)
```
---
### `!analyze -v` is misleading — please don't triage from it
The exception record in this dump has `NumberParameters: 0`, so `ExceptionInformation[0]`/`[1]` are absent, and `.ecxr` yields the **FailFast** context (`ntdll!NtRaiseException`) rather than the original fault. `!analyze -v` consequently produces a fabricated bucket:
```
Failure.Bucket: NULL_POINTER_READ_c0000005_coreclr.dll!ProcessCLRException
READ_ADDRESS: 0000000000000000
CONTEXT: 000000efccb7a550 -- all registers zero, rip=0, rsp=0
FAULTING_THREAD: ffffffff
```
None of that is real. The same applies to the `System.AccessViolationException` on the heap: its `_target` (`0x2D8B6F889B3`) and `_accessType` (1) do **not** agree with the actual faulting instruction, which is consistent with `SetExceptionAVParameters` having been handed a record with no AV parameters. All numbers below come from the `FaultingExceptionFrame`'s saved `CONTEXT` instead.
---
### Faulting context
Recovered via `.cxr` on `FaultingExceptionFrame.m_ctx`:
```
0:024> dt coreclr!FaultingExceptionFrame 000000efccb7a7f0
+0x000 __VFN_table : 0x00007ff9c3a232b8
+0x008 m_Next : 0x000000efccb7f8c8 Frame
+0x010 m_fFilterExecuted: 0n0
+0x018 m_ReturnAddress : 0x00007ff9654f2e08
+0x020 m_ctx : _CONTEXT
0:024> .cxr 000000ef`ccb7a810
rax=000002d8b6f889b3 rbx=00000298f2ff9740 rcx=00000000000a63cb
rdx=00000298f2ff97b0 rsi=0000000000000001 rdi=00000298f2ff9638
rip=00007ff9654f2e08 rsp=000000efccb7de30 rbp=0000000000000001
r8=00000298f2ff9800 r9=00000298f2ff9638 r10=00000298f2ff9f70
r11=000002988276ff70 r12=000002990f03bc44 r13=0000000000000000
r14=00007ff963be9df8 r15=00000298f2ff97b0
```
`rip` is the **resume IP** written by `AdjustContextForJITHelpers`, not the faulting instruction. The instruction there is a post-call spill reload:
```
00007ff9`654f2e01 33d2 xor edx,edx
00007ff9`654f2e03 e818d270fe call 00007ff9`63c00020 ; CORINFO_HELP_ARRADDR_ST
00007ff9`654f2e08 8bb42480000000 mov esi,dword ptr [rsp+80h] ; <- resume IP
00007ff9`654f2e0f 89735c mov dword ptr [rbx+5Ch],esi
```
### The barrier and the faulting instruction
```
0:024> u 00007ff9`63a90010 L20
63a90010 mov qword ptr [rcx],rdx ; reference store — succeeded
63a90013 mov r8,rcx
63a90016 mov rax,2D8B6F889B3h ; RegionToGeneration table (skewed)
63a90020 shr rcx,16h ; RegionShrDest = 22
63a90024 cmp byte ptr [rcx+rax],0 ; <=== FAULTS HERE
63a90028 jne 63a9002e
63a9002a rep ret
63a9002e mov r9,298A3400000h ; lower bound
63a90038 cmp rdx,r9
63a9003b jae 63a9003e
63a9003d ret
63a9003e mov r9,2D8A3000000h ; upper bound
63a90048 cmp rdx,r9
63a9004b jb 63a9004f
63a9004d rep ret
63a9004f shr rdx,16h ; RegionShrSrc = 22
63a90053 mov dl,byte ptr [rdx+rax]
63a90056 cmp dl,byte ptr [rcx+rax]
63a90059 jb 63a9005e
63a9005b rep ret
63a9005e mov rax,2D84FEC9840h ; card table
63a90068 mov ecx,r8d
63a9006b shr r8,0Bh
63a9006f shr ecx,8
63a90072 and ecx,7
63a90075 mov dl,1
63a90077 shl dl,cl
63a90079 test byte ptr [r8+rax],dl
```
The fault is pinned to `63a90024`: `rcx` is already shifted (`0xa63cb`) while `rdx` is still unshifted (`0x298f2ff97b0`, pre-`63a9004f`), and `63a90024` is the only memory access in that window.
```
fault address = rcx + rax
= 0xa63cb + 0x2d8b6f889b3
= 000002d8`b702ed7e
```
### Globals vs. patched immediates
```
0:024> dp coreclr!g_card_table L1
00007ff9`c3aad7f0 000002d8`4fec9840
0:024> dp coreclr!g_lowest_address L1
00007ff9`c3aad7e8 00000298`a3400000
0:024> dp coreclr!g_highest_address L1
00007ff9`c3aad7e0 000002d8`a3000000
0:024> dp coreclr!g_card_bundle_table L1
00007ff9`c3aad7c0 000002d8`b2ec2b28
0:024> dp coreclr!g_sw_ww_table L1
00007ff9`c3aad668 00000000`00000000
```
| Immediate | Value | Matching global | Match? |
|---|---|---|---|
| `63a9002e` lower bound | `298A3400000h` | `g_lowest_address` = `00000298a3400000` | **yes** |
| `63a9003e` upper bound | `2D8A3000000h` | `g_highest_address` = `000002d8a3000000` | **yes** |
| `63a9005e` card table | `2D84FEC9840h` | `g_card_table` = `000002d84fec9840` | **yes** |
| `63a90016` region-to-gen | `2D8B6F889B3h` | `g_region_to_generation_table` = *TODO: run `dp coreclr!g_region_to_generation_table L1`* | *TODO* |
Since `g_region_to_generation_table` and `g_card_table` are both stomped by `StompWriteBarrierResize()`, and the card-table immediate is current, a partial/torn stomp seems unlikely — pointing at the table's **memory** rather than the barrier's constant.
Unskewing to the table's real base:
```
g_lowest_address >> 22 = 0x298a3400000 >> 22 = 0xa628d
real table base = 0x2d8b6f889b3 + 0xa628d = 000002d8`b702ec40
index for destination = 0xa63cb - 0xa628d = 0x13e
fault address = 0x2d8b702ec40 + 0x13e = 000002d8`b702ed7e (matches)
```
The faulting byte is only **0x13e bytes into the table** — the table's own first page is not mapped.
### Memory state of the region-to-generation table
```
0:024> !address 000002d8`b6f889b3
Base Address: 000002d8`b30c8000
End Address: 000002d8`b702e000
Region Size: 00000000`03f66000 ( 63.398 MB)
State: 00002000 MEM_RESERVE
Protect:
Type: 00020000 MEM_PRIVATE
Allocation Base: 000002d8`a3010000
Allocation Protect: 00000004 PAGE_READWRITE
```
The GC bookkeeping allocation is based at `000002d8a3010000`, with committed pages ending at `000002d8b302e000`/`b30c8000` and ~63 MB reserved beyond. The region-to-generation table base (`0x2d8b702ec40`) sits just past the end of that reserved chunk, and the byte at index `0x13e` is unmapped.
For contrast, the card table is healthy — a live object's card resolves inside committed space:
```
Regex object 00000298a546a238 >> 11 = 0x5314a8d4
+ g_card_table (0x2d84fec9840) = 000002d8`a3014114 (committed)
```
### Escalation path
```
0f
0e ntdll!KiUserExceptionDispatch+0x2e
0d ntdll!RtlDispatchException+0x206
0c ntdll!RtlpCallVectoredHandlers+0xd6
0b coreclr!CLRVectoredExceptionHandlerShim+0x144 excep.cpp @ 7503
0a coreclr!HandleManagedFault+0xd3 excep.cpp @ 6576
09 KERNELBASE!RaiseException+0x8a
08 ntdll!RtlRaiseException+0x221 (c0000005 in args)
07 ntdll!RtlDispatchException+0x437
06 ntdll!RtlpExecuteHandlerForException+0xf
05 coreclr!ProcessCLRException+0xf129f exceptionhandling.cpp @ 1066
04 coreclr!EEPolicy::HandleFatalError+0x133 eepolicy.cpp @ 810
03 coreclr!EEPolicy::LogFatalError+0x59a eepolicy.cpp @ 571
02 coreclr!WatsonLastChance+0x1c6 excep.cpp @ 3837
01 KERNELBASE!RaiseFailFastException+0x153
00 ntdll!NtRaiseException+0x14
```
### Managed stack (thread 24, TID 0x1db8 — ThreadPool worker)
```
[FaultingExceptionFrame: 000000efccb7a7f0]
System.Text.RegularExpressions.Match..ctor(Regex, Int32, String, Int32) Match.cs @ 59
System.Text.RegularExpressions.RegexRunner.InitializeForScan(...) RegexRunner.cs @ 291
System.Text.RegularExpressions.Regex.RunSingleMatch(...) Regex.cs @ 400
System.Threading.ThreadPoolWorkQueue.Dispatch() ThreadPoolWorkQueue.cs @ 1015
System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() PortableThreadPool.WorkerThread.NonBrowser.cs @ 102
```
Nothing unusual on the application side — a plain `Regex.Match(string)` against a process-wide `static readonly Regex`, on a thread-pool worker doing a large configuration rebuild. No unsafe code, no P/Invoke, no `GC.TryStartNoGCRegion` on this path. Stack usage was ~22 KB of a 1.5 MB reservation, so no stack overflow.
`Match..ctor` had multiple native code versions (dynamic PGO active), if relevant:
```
Current CodeAddr: 00007ff9654f2d50
00007ff9654dab10 (OptimizedTier1 + Instrumented)
00007ff9654f2d50 (OptimizedTier1)
00007ff9b7ac89b0 (ReadyToRun)
```
---
### Two anomalies I could not resolve — flagging rather than explaining
**1. The destination region is not owned by any generation.**
`rcx` pre-shift was the barrier's destination; `rcx >> 22 = 0xa63cb` inverts to region base `0x298f2c00000`. That region is `MEM_COMMIT`/`PAGE_READWRITE` inside the GC's reservation (`Allocation Base 00000298a3010000`), but it is **absent from `!eeheap -gc`** and rejected by `!GCWhere`:
```
0:024> !GCWhere 00000298`f2ff9740
Address 298f2ff9740 not found in the managed heap.
0:024> !verifyobj 00000298`f2ff9740
0x00 ObjectNotOnTheHeap Tried to validate 298f2ff9740 but its address was not on any segment.
```
Neighbouring 4 MB regions at `0x298f2000000`, `0x298f2800000`, `0x298f3000000` are all present in `!eeheap -gc`; `0x298f2400000` and `0x298f2c00000` are not. Note `0x298f2c00000` **is** inside `[g_lowest_address, g_highest_address]`, so it should be covered by the region map.
Despite that, the object at `0x298f2ff9740` is a well-formed `Match`:
```
0:024> dq 00000298`f2ff9740 L10
+0x00 00007ff9`64282e18 ; Match MethodTable (confirmed by !ip2md)
+0x08 00000299`0f03bc38 ; Text — the input string
+0x18 00000299`0f03bdb0 ; _caps
+0x40 00000298`a546a238 ; _regex — the actual Regex instance
+0x48 00000299`0f03bdf0 ; _matches
+0x50 00000299`0f03bdd0 ; _matchcount
```
All of those references are valid and land in the live gen0 region (`02990f000028`–`02990f3ad9f8`).
`!verifyheap` reported **12,557,451 objects verified, 0 errors** — but it would not have walked this object, since its region isn't enumerated.
**2. `rcx` does not correspond to the array element address.**
The resume IP follows the `CORINFO_HELP_ARRADDR_ST` call at `654f2e03`, whose array operand was `_matches = 0x2990f03bdf0`:
```
&_matches[0] = 0x2990f03be00 → >> 22 = 0xa643c
rbx + 0x48 = 0x298f2ff9788 → >> 22 = 0xa63cb (= observed rcx)
```
So the shifted `rcx` matches a destination inside the `Match`'s own region rather than the gen0 array element. I cannot reconcile that with the resume IP and am reporting it as observed.
---
### Relationship to #123490 / #131310
[#131310](https://github.com/dotnet/runtime/pull/131310) (fixing [#123490](https://github.com/dotnet/runtime/issues/123490)) describes, for Workstation-segments on x86:
> A compacting foreground GC creates expansion segment S and marks its mark-array slice committed on the current array → before S is linked to generation lists, a card-table grow fires and allocates a new, larger mark array → `commit_new_mark_array` walks only the generation segment lists, missing the unlinked S → S's pages are never committed on the new array while its `heap_segment_flags_ma_committed` flag remains set → a later BGC trusts the stale flag and crashes writing to uncommitted memory.
The failure mode here is the same shape — a bookkeeping-table grow leaving a sub-table's pages uncommitted, with the fault surfacing later on reserved memory — but on a different path: **x64, regions (`USE_REGIONS`), the region-to-generation map, consumed by the JIT write barrier** rather than the mark array consumed by BGC marking. Filing separately in case the regions path needs its own fix; happy to have it closed as a duplicate if #131310's root cause covers it.
### Repro
None. Single occurrence after 15.2 days of continuous operation in a production service. Not reproducible on demand. The workload that was running is a large configuration rebuild that allocates heavily (~1.45 GB live heap), so it crosses bookkeeping-table growth boundaries repeatedly.
### What would help
1. Confirmation of whether the barrier's region-to-generation immediate is expected to match `g_region_to_generation_table` at all times, and whether the region map is expected to be fully committed for `[g_lowest_address, g_highest_address]`.
2. Whether a region can legitimately be inside `[g_lowest_address, g_highest_address]`, hold live objects, and be absent from `!eeheap -gc`/`!GCWhere` — or whether that is itself evidence of the bug.
3. Whether #131310's root cause has a `USE_REGIONS` analogue, and if a fix would be backported to 8.0.
### Reproduction Steps
No repro
### Expected behavior
App should not crash
### Actual behavior
App crashes
### Regression?
_No response_
### Known Workarounds
None
### Configuration
_No response_
### Other information
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.