dotnet / dotnet/runtime

[EventPipe] AV at EventPipeInternal_GetNextEvent

Open
#122,630 2 comments 0 reactions 0 assignees View on GitHub
area-Tracing-coreclr
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Description

Some users that have been leveraging an EventListener to listen to basic GC events have been experiencing crashes due to an AV at https://github.com/dotnet/runtime/blob/b2969ed9b22ed1dd7cc6619ded58831111e05354/src/coreclr/vm/eventpipeinternal.cpp#L251

### Reproduction Steps

Unclear how to repro. The users were using .NET 8 and seem to just be using a standard EventListener to listen to events. The users mentioned that this crash seems to only occur in machines that have had their OS patched with kernel upgrades.

### Expected behavior

EventPipe doesn't crash the runtime and doesn't return an invalid EventPipeEventInstance.

### Actual behavior

Crashes due to AV

### Regression?

_No response_

### Known Workarounds

Disable EventListeners, use alternative APIs to grab the data held by the events, like GCMemoryInfo or other GC APIs for GC events.

### Configuration

.NET 8, Linux-x64, kernel: 5.15.0-140-fips, AzureLinux3.0

### Other information

While investigating the crash dump, we've observed that there is an EventPipeBuffer with it's [buffer region](https://github.com/dotnet/runtime/blob/b2969ed9b22ed1dd7cc6619ded58831111e05354/src/native/eventpipe/ep-buffer.h#L49-L53) all 0'd out, even though the EventPipeBuffer still needed its first EventPipeEventInstance read from its buffer region. (below `current_read_event` is at `buffer`)
```
0:224> ?? ((EventPipeSession*)sessionID)->buffer_manager->current_buffer
EventPipeBuffer * 0x00007f5e`280bfae0
+0x000 creation_timestamp : 0n1876226656953887
+0x008 writer_thread : 0x00007f5e`28000b40 _EventPipeThread
+0x010 buffer : 0x00007fbe`fcd89000 -> 0 ''
+0x018 current : 0x00007fbe`fcd89f78 -> 0 ''
+0x020 limit : 0x00007fbe`fcda2000 -> ??
+0x028 current_read_event : 0x00007fbe`fcd89000 _EventPipeEventInstance
+0x030 prev_buffer : (null)
+0x038 next_buffer : 0x00007f5e`280fc890 _EventPipeBuffer
+0x040 state : 1
+0x044 event_sequence_number : 0x2edc13
```

This is unexpected, as buffer regions should only be cleared when
1. [Buffer allocation failed](https://github.com/dotnet/runtime/blob/b2969ed9b22ed1dd7cc6619ded58831111e05354/src/native/eventpipe/ep-buffer.c#L49) - This buffer shouldn't have been added to the buffer list
2. [Buffer manager failed to add the buffer to list](https://github.com/dotnet/runtime/blob/b2969ed9b22ed1dd7cc6619ded58831111e05354/src/native/eventpipe/ep-buffer-manager.c#L546) - The buffer wouldn't have been added to the buffer list
3. [While advancing to a non-empty buffer, the current buffer's read event is NULL](https://github.com/dotnet/runtime/blob/b2969ed9b22ed1dd7cc6619ded58831111e05354/src/native/eventpipe/ep-buffer-manager.c#L735). From the dump, it's not NULL and is still the first EventPipeEventInstance.

Moreover, the thread that was logging to this buffer had two buffers in its list, the readable `current_buffer` above, and another writable buffer whose event sequence number was `0x2edc1c`, indicating 9 events (0x2edc1c - 0x2edc13) being written in `current_buffer` above. The writable buffer also had valid non-zeroed EventPipeEventInstances in its buffer region.

With insufficient diagnostics to determine what exactly led to the EventPipeBuffer's buffer region being 0'd out, coupled with the dump suggesting a scenario that doesn't seem consistent with EventPipe logic, it's difficult to tell if this crash was really due to a race condition leading to use-after-free, or some external process corrupting memory.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.