dotnet / dotnet/extensions

HttpServiceEndpointResolver leaked per HTTP handler build in Microsoft.Extensions.ServiceDiscovery

Open
#7,670 1 comment 0 reactions 0 assignees View on GitHub
bug untriaged
Dominant language
C#
Stars
3.2k
Forks
894
Avg merge
1d 12h
Merged PRs (30d)
23

Description

### Description

AddServiceDiscovery allocates a new HttpServiceEndpointResolver for every HTTP message handler that gets built. Because HttpClientFactory rebuilds handler chains on the normal handler-lifetime rotation (default 2 minutes), a fresh resolver is created periodically for the entire life of the process.

HttpServiceEndpointResolver is IAsyncDisposable and owns endpoint-refresh timers plus configuration change-token subscriptions, but ResolvingHttpDelegatingHandler wraps it and never disposes it. Each orphaned resolver stays rooted (by the runtime timer queue and the configuration root) and never becomes eligible for collection.

Its live timers keep firing, so the leak manifests as a slow, uptime-correlated CPU climb (and a steadily growing object count), not a memory blowup. I found this watching a new minimal service slowly but surely grow in CPU utilization over the course of days. Because the churn is driven by handler rotation rather than request volume, it does not reproduce under short load tests, but rather only under sustained uptime.

### Reproduction Steps

```csharp
using System.Reflection;
using Microsoft.Extensions.DependencyInjection;

var services = new ServiceCollection();
services.AddHttpClient("svc")
.AddServiceDiscovery()
.SetHandlerLifetime(TimeSpan.FromSeconds(1)); // short so the chain rebuilds quickly
await using var provider = services.BuildServiceProvider();

var factory = provider.GetRequiredService();
var resolvers = new HashSet(ReferenceEqualityComparer.Instance);

for (var i = 0; i < 4; i++)
{
var handler = factory.CreateHandler("svc"); // rebuilt after each lifetime expiry
var resolving = FindHandler(handler, "ResolvingHttpDelegatingHandler");
var resolver = resolving.GetType()
.GetField("_resolver", BindingFlags.Instance | BindingFlags.NonPublic)!
.GetValue(resolving)!;
resolvers.Add(resolver);
Console.WriteLine($"iteration {i}: distinct resolvers so far = {resolvers.Count}");
await Task.Delay(TimeSpan.FromMilliseconds(1200)); // let the handler lifetime expire
}

Console.WriteLine($"TOTAL distinct HttpServiceEndpointResolver instances: {resolvers.Count}");

static HttpMessageHandler FindHandler(HttpMessageHandler handler, string typeName)
{
for (var c = handler; c is not null; c = (c as DelegatingHandler)?.InnerHandler)
{
if (c.GetType().Name == typeName)
{
return c;
}
}

throw new InvalidOperationException($"{typeName} not found in handler chain");
}
```
Output
```
iteration 0: distinct resolvers so far = 1
iteration 1: distinct resolvers so far = 2
iteration 2: distinct resolvers so far = 3
iteration 3: distinct resolvers so far = 4
TOTAL distinct HttpServiceEndpointResolver instances: 4
```

### Expected behavior

Number of `HttpServiceEndpointResolver`s stays constant or constrained

### Actual behavior

Number of `HttpServiceEndpointResolver`s grows over time

### Regression?

Exists in 10.6.0 and 10.8.0 - Unknown if it exists prior.

### Known Workarounds

Pin the handler lifetime so the chain (and its resolver) isn't rebuilt: ConfigureHttpClientDefaults(b => b.SetHandlerLifetime(Timeout.InfiniteTimeSpan)), paired with SocketsHttpHandler.PooledConnectionLifetime to preserve DNS refresh.

### Configuration

Dotnet 10.0.302
Azure container apps
Not specific to that config

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing AddServiceDiscovery into the construction and disposal path for ResolvingHttpDelegatingHandler and HttpServiceEndpointResolver. Run the supplied short handler-lifetime reproduction and verify that retired resolvers no longer remain rooted, their timers stop, and the resolver count stays constant or constrained.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.