dotnet / dotnet/extensions

ServiceDiscovery: each ServiceEndpointWatcher lifecycle permanently leaks a change-token registration on IConfiguration

Open
#7,673 0 comments 0 reactions 0 assignees View on GitHub
bug untriaged
Dominant language
C#
Stars
3.2k
Forks
894
Avg merge
1d 12h
Merged PRs (30d)
23

Description

### Description

`ServiceEndpointResolver` evicts a service's `ServiceEndpointWatcher` when the name has not been resolved recently, using a cleanup timer with a 10 second period. The next resolution of that name creates a fresh watcher, which performs an initial refresh. Every one of those refreshes permanently adds a callback registration to the application's root `IConfiguration` reload token, and the registration is never released, neither when the watcher is disposed nor when it is evicted.

An application resolving a service name less often than roughly every 10 seconds therefore leaks one `CompositeChangeToken`, one `CancellationTokenSource`, two `CancellationTokenSource+CallbackNode`s and their supporting lists and arrays on every resolution, all rooted from the configuration reload token for the lifetime of the process. Resolving the same name more frequently avoids the leak, because the entry stays marked as recently used and its watcher is never evicted.

`ServiceEndpointWatcher` disposes its previous change-token registration on each refresh and again in `DisposeAsync`. Disposing a registration on a `CompositeChangeToken` releases the consumer's callback but leaves the composite's proxy registrations on its inner tokens in place; `CompositeChangeToken` releases those only when it fires, and a configuration reload token in a typical deployment never fires. `ConfigurationServiceEndpointProvider` contributes that inner token on every populate, on both of its branches.

### Reproduction Steps

```
dotnet new console
dotnet add package Microsoft.Extensions.ServiceDiscovery --version 10.6.0
dotnet add package Microsoft.Extensions.Hosting
```

```csharp
using System.Reflection;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.ServiceDiscovery;

static long Registrations(IConfigurationRoot root)
{
var token = root.GetReloadToken();
var cts = (CancellationTokenSource?)token.GetType()
.GetField("_cts", BindingFlags.NonPublic | BindingFlags.Instance)!.GetValue(token);
var regs = typeof(CancellationTokenSource)
.GetField("_registrations", BindingFlags.NonPublic | BindingFlags.Instance)!.GetValue(cts!);
if (regs is null) return 0;
return (long)regs.GetType()
.GetField("NextAvailableId", BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance)!
.GetValue(regs)!;
}

var builder = Host.CreateApplicationBuilder();
builder.Services.AddServiceDiscovery();

using var host = builder.Build();
var root = (IConfigurationRoot)builder.Configuration;
var resolver = host.Services.GetRequiredService();

for (var i = 1; i <= 8; i++)
{
await resolver.GetEndpointsAsync("http://some-service", CancellationToken.None);
Console.WriteLine($"resolution {i}: {Registrations(root)} registrations");
await Task.Delay(TimeSpan.FromSeconds(25)); // longer than the 10 second cleanup period
}
```

Output, one registration added per resolution and never released:

```
resolution 1: 5
resolution 2: 6
resolution 3: 7
resolution 4: 8
resolution 5: 9
resolution 6: 10
resolution 7: 11
resolution 8: 12
```

### Expected behavior

Registrations on `IConfiguration`'s reload token stay bounded. Creating and discarding N watchers for a service name leaves O(1) registrations, not O(N).

### Actual behavior

Registrations grow by one per watcher lifecycle and are never released.

In a .NET 10 service resolving 9 service names from a background loop on a 30 second timer, so that every name is evicted between uses and re-created on each pass, this ran at **~21 leaked registrations per minute**, with generation 2 climbing **~15.7 MiB/day**, zero drawdown across 22 gen2 collections and flat fragmentation.

A dump after 63 hours of uptime showed about **41 MB, roughly half the managed heap**, in 77,182 `CompositeChangeToken`, 152,782 `CancellationTokenSource+CallbackNode`, 76,463 `CancellationTokenSource`, 76,424 `CancellationTokenSource+Registrations`, and ~76,350 each of `IChangeToken[]`, `IDisposable[]`, `List` and `List`.

All of it hangs off a single `CancellationTokenSource`, `ConfigurationManager._changeToken._cts`, whose `Registrations.NextAvailableId` read **76,339** with the whole callback list still linked. Each leaked composite's own CTS reads `Callbacks = null, NextAvailableId = 2`: the watcher registered once and disposed correctly, and what is retained is the composite's proxy registration on its inner token.

Allocation stack, from `dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:0x1:5`:

```
System.Collections.Generic.List`1[System.__Canon].AddWithResize(!0)
Microsoft.Extensions.ServiceDiscovery.Configuration.ConfigurationServiceEndpointProvider.PopulateAsync(IServiceEndpointBuilder, CancellationToken)
Microsoft.Extensions.ServiceDiscovery.ServiceEndpointWatcher+d__23.MoveNext()
System.Threading.ThreadPoolWorkQueue.Dispatch()
System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
```

### Regression?

_No response_

### Known Workarounds

_No response_

### Configuration

- `Microsoft.Extensions.ServiceDiscovery` **10.6.0**, base package only, so the registered providers are Configuration and PassThrough.
- .NET **10.0.10**. Observed in a Linux container (`mcr.microsoft.com/dotnet/aspnet:10.0`, x64) and reproduced on Windows with the console app above.

### Other information

The mechanism, with the source involved.

**1. The resolver evicts idle watchers every 10 seconds**, `src/Libraries/Microsoft.Extensions.ServiceDiscovery/ServiceEndpointResolver.cs`:

```csharp
private static readonly TimerCallback s_cleanupCallback = s => ((ServiceEndpointResolver)s!).CleanupResolvers();
private static readonly TimeSpan s_cleanupPeriod = TimeSpan.FromSeconds(10);
...
if (resolver.CanExpire() && _resolvers.TryRemove(name, out var _))
```

A name resolved less often than that gets a new `ServiceEndpointWatcher`, and therefore a fresh initial refresh, on its next resolution.

**2. The configuration provider contributes the configuration root's reload token on every populate**, on both branches, `Configuration/ConfigurationServiceEndpointProvider.cs`:

```csharp
var section = _configuration.GetSection(_options.Value.SectionName).GetSection(_serviceName);
if (!section.Exists())
{
endpoints.AddChangeToken(_configuration.GetReloadToken());
Log.ServiceConfigurationNotFound(_logger, _serviceName, $"{_options.Value.SectionName}:{_serviceName}");
return default;
}

endpoints.AddChangeToken(section.GetReloadToken());
```

Both are the same object, since `ConfigurationSection.GetReloadToken()` delegates to the root (`dotnet/runtime`, `Microsoft.Extensions.Configuration/src/ConfigurationSection.cs`):

```csharp
public IChangeToken GetReloadToken() => _root.GetReloadToken();
```

**3. The builder wraps the tokens in a composite unconditionally**, `ServiceEndpointBuilder.cs`:

```csharp
public ServiceEndpointSource Build()
{
return new ServiceEndpointSource(_endpoints, new CompositeChangeToken(_changeTokens), _features);
}
```

**4. The watcher registers on that composite**, `ServiceEndpointWatcher.RefreshAsyncInternal`:

```csharp
else if (endpoints.ChangeToken.ActiveChangeCallbacks)
{
_changeTokenRegistration = endpoints.ChangeToken.RegisterChangeCallback(
static state => _ = ((ServiceEndpointWatcher)state!).RefreshAsync(force: false), this);
}
```

`ConfigurationReloadToken.ActiveChangeCallbacks` is always `true`, so this branch is always taken and the polling timer is always disabled.

**5. `CompositeChangeToken` releases its inner registrations only when it fires**, `dotnet/runtime`, `Microsoft.Extensions.Primitives/src/CompositeChangeToken.cs`. `RegisterChangeCallback` calls `EnsureCallbacksInitialized()`:

```csharp
IDisposable disposable = ChangeTokens[i].RegisterChangeCallback(_onChangeDelegate, this);
_disposables.Add(disposable);
```

`_disposables` is disposed only inside the static `OnChange` handler, and the type has no public `Dispose`.

The watcher is created, registers on composite C, and C registers a proxy on the configuration reload token. The watcher is later evicted; `DisposeAsync` disposes the watcher's registration on C, which leaves C's proxy registration on the reload token in place. C, its CTS, its callback nodes and its lists stay rooted from the application's configuration, and the cycle repeats on the next resolution.

Contributor guide

Open the contributing guide

Research direction

Run the supplied console reproduction first, then trace watcher eviction in src/Libraries/Microsoft.Extensions.ServiceDiscovery/ServiceEndpointResolver.cs through ServiceEndpointWatcher.RefreshAsyncInternal. Read Configuration/ConfigurationServiceEndpointProvider.cs, ServiceEndpointBuilder.cs, and CompositeChangeToken.cs to follow registration ownership. Done means repeated resolutions after the 10-second cleanup period leave configuration reload-token registrations bounded rather than growing per watcher.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.