microsoft / microsoft/aspire

Deadlock in ServiceDiscovery library leading to thread pool starvation

Open
#9,957 3 comments 2 reactions 0 assignees View on GitHub
area-service-discovery
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

A deadlock can occur between `Microsoft.Extensions.ServiceDiscovery.dll!Microsoft.Extensions.ServiceDiscovery.ServiceEndpointWatcher.RefreshAsyncInternal() Line 141` and `Microsoft.Extensions.ServiceDiscovery.dll!Microsoft.Extensions.ServiceDiscovery.ServiceEndpointWatcher.RefreshAsync(bool force) Line 96`, leading to thread pool starvation.

On 5/30, we noticed that one of our reverse proxy application nodes had stopped servicing requests. This application uses YARP to route traffic to our backend and apply some custom authentication logic (among other things). We’re using Aspire for local development, and in production we have an explicit `Services` section in our appsettings.json file to configure service discovery using `Microsoft.Extensions.ServiceDiscovery`.

We took a memory dump before restarting. This dump shows 4435 worker threads sitting at `Microsoft.Extensions.ServiceDiscovery.dll!Microsoft.Extensions.ServiceDiscovery.ServiceEndpointWatcher.RefreshAsync` and 1 worker thread at `Microsoft.Extensions.ServiceDiscovery.dll!Microsoft.Extensions.ServiceDiscovery.ServiceEndpointWatcher.RefreshAsyncInternal`.

My analysis: For the worker thread at RefreshAsyncInternal, it is stuck on [line 141](https://github.com/dotnet/aspire/blob/2a8f48ea5811f317a26405eb315aa315cc9e3cea/src/Microsoft.Extensions.ServiceDiscovery/ServiceEndpointWatcher.cs#L141), blocking inside of a lock waiting on `_changeTokenRegistration?.Dispose()` to complete. On [line 162](https://github.com/dotnet/aspire/blob/2a8f48ea5811f317a26405eb315aa315cc9e3cea/src/Microsoft.Extensions.ServiceDiscovery/ServiceEndpointWatcher.cs#L162), you can see that `_changeTokenRegistration` is given a value of a change callback registration which itself calls `RefreshAsync`, so I suspect that there is a race condition here where [the lock on line 138](https://github.com/dotnet/aspire/blob/2a8f48ea5811f317a26405eb315aa315cc9e3cea/src/Microsoft.Extensions.ServiceDiscovery/ServiceEndpointWatcher.cs#L138) was entered just before the lock was entered inside of that callback, and now `_changeTokenRegistration.Dispose()` is waiting for that callback to complete before continuing, which will never happen. I’m not sure how to track down which exact task is blocking the dispose.

I can’t provide the memory dump itself as it was taken from a production environment but I am happy to answer questions or whatever else is needed.

### Expected Behavior

No deadlock.

### Steps To Reproduce

Unsure how to reproduce, seems to be a race condition that is unlikely to come up. Have only had it happen once over the course of a few months.

### Exceptions (if any)

None.

### .NET Version info

Here’s the output of dotnet --info from today. At the time of the bug, we were still running .NET 9.0.4.

```
Host:
Version: 9.0.5
Architecture: x64
Commit: e36e4d1a8f
RID: win-x64

.NET SDKs installed:
No SDKs were found.

.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 8.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
Not set

global.json file:
Not found

Learn more:
https://aka.ms/dotnet/info

Download .NET:
https://aka.ms/dotnet/download
```

### Anything else?

At the time of the issue, we were running .NET 9.0.4 and [ASP.NET](http://asp.net/) Core 9.0.4, confirmed in the module list in the memory dump.

The version of Microsoft.Extensions.ServiceDiscovery we have installed is 9.1.0. An excerpt from our Directory.Packages.props file:

```xml

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.