dotnet / dotnet/aspnetcore

NRE in Kestrel Telemetry EventSources

Open
#44,817 3 comments 0 reactions 0 assignees View on GitHub
area-networking feature-kestrel
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 10h
Merged PRs (30d)
281

Description

Mirror of https://github.com/dotnet/runtime/issues/77434

`EventSource.IsEnabled()` may start returning `true` before the `OnEventCommand` completes.
We initialize shared `EventCounter` instances in `OnEventCommand` that we then use from instrumented code paths.

If the `EventSource` is initialized before being enabled, and the instance is accessed from multiple threads, a thread could see `IsEnabled() == true` and then read a `null` `EventCounter` instance.
This race condition only appears once per process.

I believe this is the cause of an exception [YARP hit in CI](https://dev.azure.com/dnceng-public/public/_build/results?buildId=61231&view=ms.vss-test-web.build-test-results-tab&runId=1256708&resultId=100101&paneView=debug):
```
System.NullReferenceException : Object reference not set to an instance of an object.
at System.Net.Http.HttpTelemetry.Http11RequestLeftQueue(Double timeOnQueueMilliseconds)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
```

We should fix [these cases](https://grep.app/search?q=%3F%3F%3D%20new%20EventCounter%28&filter[repo.pattern][0]=dotnet).

cc: @Tratcher aspnetcore has 2 such cases as well:
https://github.com/dotnet/aspnetcore/blob/1e8fe7e4ba93935ebc10aee6ef2a4d2b3b0c05d9/src/Middleware/ConcurrencyLimiter/src/ConcurrencyLimiterEventSource.cs#L47
https://github.com/dotnet/aspnetcore/blob/1e8fe7e4ba93935ebc10aee6ef2a4d2b3b0c05d9/src/SignalR/common/Http.Connections/src/Internal/HttpConnectionsEventSource.cs#L46

An internal customer hit something similar:
![MicrosoftTeams-image (1)](https://user-images.githubusercontent.com/1821173/199293026-22c148bf-82a0-4160-b13e-cbc44a3d72c8.png)

![MicrosoftTeams-image (2)](https://user-images.githubusercontent.com/1821173/199293037-e1708dde-9124-43cf-adbb-df9a9f204555.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.