HttpServiceEndpointResolver leaked per HTTP handler build in Microsoft.Extensions.ServiceDiscovery
- Dominant language
- C#
- Stars
- 3.2k
- Forks
- 894
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
### Description
AddServiceDiscovery allocates a new HttpServiceEndpointResolver for every HTTP message handler that gets built. Because HttpClientFactory rebuilds handler chains on the normal handler-lifetime rotation (default 2 minutes), a fresh resolver is created periodically for the entire life of the process.
HttpServiceEndpointResolver is IAsyncDisposable and owns endpoint-refresh timers plus configuration change-token subscriptions, but ResolvingHttpDelegatingHandler wraps it and never disposes it. Each orphaned resolver stays rooted (by the runtime timer queue and the configuration root) and never becomes eligible for collection.
Its live timers keep firing, so the leak manifests as a slow, uptime-correlated CPU climb (and a steadily growing object count), not a memory blowup. I found this watching a new minimal service slowly but surely grow in CPU utilization over the course of days. Because the churn is driven by handler rotation rather than request volume, it does not reproduce under short load tests, but rather only under sustained uptime.
### Reproduction Steps
```csharp
using System.Reflection;
using Microsoft.Extensions.DependencyInjection;
var services = new ServiceCollection();
services.AddHttpClient("svc")
.AddServiceDiscovery()
.SetHandlerLifetime(TimeSpan.FromSeconds(1)); // short so the chain rebuilds quickly
await using var provider = services.BuildServiceProvider();
var factory = provider.GetRequiredService();
var resolvers = new HashSet(ReferenceEqualityComparer.Instance);
for (var i = 0; i < 4; i++)
{
var handler = factory.CreateHandler("svc"); // rebuilt after each lifetime expiry
var resolving = FindHandler(handler, "ResolvingHttpDelegatingHandler");
var resolver = resolving.GetType()
.GetField("_resolver", BindingFlags.Instance | BindingFlags.NonPublic)!
.GetValue(resolving)!;
resolvers.Add(resolver);
Console.WriteLine($"iteration {i}: distinct resolvers so far = {resolvers.Count}");
await Task.Delay(TimeSpan.FromMilliseconds(1200)); // let the handler lifetime expire
}
Console.WriteLine($"TOTAL distinct HttpServiceEndpointResolver instances: {resolvers.Count}");
static HttpMessageHandler FindHandler(HttpMessageHandler handler, string typeName)
{
for (var c = handler; c is not null; c = (c as DelegatingHandler)?.InnerHandler)
{
if (c.GetType().Name == typeName)
{
return c;
}
}
throw new InvalidOperationException($"{typeName} not found in handler chain");
}
```
Output
```
iteration 0: distinct resolvers so far = 1
iteration 1: distinct resolvers so far = 2
iteration 2: distinct resolvers so far = 3
iteration 3: distinct resolvers so far = 4
TOTAL distinct HttpServiceEndpointResolver instances: 4
```
### Expected behavior
Number of `HttpServiceEndpointResolver`s stays constant or constrained
### Actual behavior
Number of `HttpServiceEndpointResolver`s grows over time
### Regression?
Exists in 10.6.0 and 10.8.0 - Unknown if it exists prior.
### Known Workarounds
Pin the handler lifetime so the chain (and its resolver) isn't rebuilt: ConfigureHttpClientDefaults(b => b.SetHandlerLifetime(Timeout.InfiniteTimeSpan)), paired with SocketsHttpHandler.PooledConnectionLifetime to preserve DNS refresh.
### Configuration
Dotnet 10.0.302
Azure container apps
Not specific to that config
### Other information
_No response_
Contributor guide
Research direction
Start by tracing AddServiceDiscovery into the construction and disposal path for ResolvingHttpDelegatingHandler and HttpServiceEndpointResolver. Run the supplied short handler-lifetime reproduction and verify that retired resolvers no longer remain rooted, their timers stop, and the resolver count stays constant or constrained.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100