[EventPipe] AV at EventPipeInternal_GetNextEvent
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- PR merge metrics
- PR metrics pending
Description
### Description
Some users that have been leveraging an EventListener to listen to basic GC events have been experiencing crashes due to an AV at https://github.com/dotnet/runtime/blob/b2969ed9b22ed1dd7cc6619ded58831111e05354/src/coreclr/vm/eventpipeinternal.cpp#L251
### Reproduction Steps
Unclear how to repro. The users were using .NET 8 and seem to just be using a standard EventListener to listen to events. The users mentioned that this crash seems to only occur in machines that have had their OS patched with kernel upgrades.
### Expected behavior
EventPipe doesn't crash the runtime and doesn't return an invalid EventPipeEventInstance.
### Actual behavior
Crashes due to AV
### Regression?
_No response_
### Known Workarounds
Disable EventListeners, use alternative APIs to grab the data held by the events, like GCMemoryInfo or other GC APIs for GC events.
### Configuration
.NET 8, Linux-x64, kernel: 5.15.0-140-fips, AzureLinux3.0
### Other information
While investigating the crash dump, we've observed that there is an EventPipeBuffer with it's [buffer region](https://github.com/dotnet/runtime/blob/b2969ed9b22ed1dd7cc6619ded58831111e05354/src/native/eventpipe/ep-buffer.h#L49-L53) all 0'd out, even though the EventPipeBuffer still needed its first EventPipeEventInstance read from its buffer region. (below `current_read_event` is at `buffer`)
```
0:224> ?? ((EventPipeSession*)sessionID)->buffer_manager->current_buffer
EventPipeBuffer * 0x00007f5e`280bfae0
+0x000 creation_timestamp : 0n1876226656953887
+0x008 writer_thread : 0x00007f5e`28000b40 _EventPipeThread
+0x010 buffer : 0x00007fbe`fcd89000 -> 0 ''
+0x018 current : 0x00007fbe`fcd89f78 -> 0 ''
+0x020 limit : 0x00007fbe`fcda2000 -> ??
+0x028 current_read_event : 0x00007fbe`fcd89000 _EventPipeEventInstance
+0x030 prev_buffer : (null)
+0x038 next_buffer : 0x00007f5e`280fc890 _EventPipeBuffer
+0x040 state : 1
+0x044 event_sequence_number : 0x2edc13
```
This is unexpected, as buffer regions should only be cleared when
1. [Buffer allocation failed](https://github.com/dotnet/runtime/blob/b2969ed9b22ed1dd7cc6619ded58831111e05354/src/native/eventpipe/ep-buffer.c#L49) - This buffer shouldn't have been added to the buffer list
2. [Buffer manager failed to add the buffer to list](https://github.com/dotnet/runtime/blob/b2969ed9b22ed1dd7cc6619ded58831111e05354/src/native/eventpipe/ep-buffer-manager.c#L546) - The buffer wouldn't have been added to the buffer list
3. [While advancing to a non-empty buffer, the current buffer's read event is NULL](https://github.com/dotnet/runtime/blob/b2969ed9b22ed1dd7cc6619ded58831111e05354/src/native/eventpipe/ep-buffer-manager.c#L735). From the dump, it's not NULL and is still the first EventPipeEventInstance.
Moreover, the thread that was logging to this buffer had two buffers in its list, the readable `current_buffer` above, and another writable buffer whose event sequence number was `0x2edc1c`, indicating 9 events (0x2edc1c - 0x2edc13) being written in `current_buffer` above. The writable buffer also had valid non-zeroed EventPipeEventInstances in its buffer region.
With insufficient diagnostics to determine what exactly led to the EventPipeBuffer's buffer region being 0'd out, coupled with the dump suggesting a scenario that doesn't seem consistent with EventPipe logic, it's difficult to tell if this crash was really due to a race condition leading to use-after-free, or some external process corrupting memory.
Contributor guide
Assessment
This issue has not been assessed yet.