microsoft / microsoft/aspire

Service discovery: missed endpoint refreshes when providers signal during ServiceEndpointWatcher refresh (upstream Microsoft.Extensions.ServiceDiscovery)

Open
#16,400 4 comments 0 reactions 0 assignees View on GitHub
area-service-discovery
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

## Summary

Apps that use .NET service discovery (including Aspire via `AddServiceDiscovery()` / service defaults) can **miss endpoint updates** when a provider raises change notifications (`IChangeToken` / reload-style signals) while the internal `ServiceEndpointWatcher` is already running a refresh, or during the window after the watcher disposes the previous change-token registration and before it completes `PopulateAsync` for all providers and registers a new callback.

The behavior comes from **`Microsoft.Extensions.ServiceDiscovery`** in **[dotnet/extensions](https://github.com/dotnet/extensions)** (not Aspire-only code). Aspire-hosted workloads are a reasonable place to track user-visible impact and coordination with the owning team.

## Expected behavior

Every provider signal that means “endpoints may have changed” should **eventually** result in another resolution pass (possibly coalesced), not be dropped silently.

## Actual behavior

Under load from **multiple sequential `PopulateAsync` calls** and **overlapping change signals**, the resolver can settle on **fewer refreshes** than reload events. Symptoms can include **stale endpoints** or **fewer `PopulateAsync` cycles** than reload events.

## Upstream implementation (for maintainers)

Primary file: [`ServiceEndpointWatcher.cs`](https://github.com/dotnet/extensions/blob/main/src/Libraries/Microsoft.Extensions.ServiceDiscovery/ServiceEndpointWatcher.cs)

Notes (high level):

1. **Callback gap:** `RefreshAsyncInternal` disposes the prior `RegisterChangeCallback` registration, awaits each provider’s `PopulateAsync`, then registers a callback on the new composite change token after `Build()`. Signals between dispose and re-register can be missed.

2. **No pending refresh while in flight:** Overlapping notifications during an in-progress refresh may not enqueue another refresh.

3. **New composite each refresh:** Each successful `Build()` wraps new token instances; signals on older generations may not appear in the next composite (validate alongside watcher lifecycle).

## Related (distinct) upstream issue

- **[dotnet/extensions#7013](https://github.com/dotnet/extensions/issues/7013)** — `RefreshPeriod` / polling with mixed providers and `ActiveChangeCallbacks`. Same component, different concern than “lost notifications during refresh.”

## Repro

1. Install a recent **.NET 10** SDK.

2. `dotnet new console -n SdRepro -f net10.0`

3. `cd SdRepro`

4. `dotnet add package Microsoft.Extensions.Hosting`

5. `dotnet add package Microsoft.Extensions.ServiceDiscovery`
Use a package version aligned with your SDK / Aspire stack (same major as your dependencies).

6. Replace `Program.cs` with the contents below (entire file).

7. `dotnet run` — let it run for the built-in **12 seconds**.

8. Compare **`timerSignals`** to **`PopulateAsync` (fast provider)**. If **`timerSignals` is much larger**, reload notifications are outpacing observed refreshes. The repro sets `RefreshPeriod = Timeout.InfiniteTimeSpan` so **polling does not mask** the gap.

## Program.cs

```csharp
using System.Diagnostics.CodeAnalysis;
using System.Net;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Primitives;
using Microsoft.Extensions.ServiceDiscovery;

HostApplicationBuilder builder = Host.CreateApplicationBuilder(args);
builder.Logging.ClearProviders();

builder.Services.AddServiceDiscoveryCore(options =>
{
// Rely on change tokens, not polling, so the gap is visible.
options.RefreshPeriod = Timeout.InfiniteTimeSpan;
});

builder.Services.AddSingleton();
builder.Services.AddSingleton();

using IHost host = builder.Build();

ServiceEndpointResolver resolver = host.Services.GetRequiredService();
const string serviceName = "http://repro-service";

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(12));
CancellationToken stopping = cts.Token;

var resolveLoop = Task.Run(async () =>
{
while (!stopping.IsCancellationRequested)
{
_ = await resolver.GetEndpointsAsync(serviceName, stopping).ConfigureAwait(false);
await Task.Delay(25, stopping).ConfigureAwait(false);
}
}, stopping);

var report = Task.Run(async () =>
{
while (!stopping.IsCancellationRequested)
{
Console.WriteLine(
"[{0:O}] timerSignals={1} populateFast={2} populateSlow={3}",
DateTimeOffset.UtcNow,
FastSignalFactory.TimerSignals,
FastSignalFactory.PopulateCount,
SlowNoopFactory.PopulateCount);
await Task.Delay(1000, stopping).ConfigureAwait(false);
}
}, stopping);

try
{
await Task.WhenAll(resolveLoop, report).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
// expected
}

Console.WriteLine();
Console.WriteLine("--- totals ---");
Console.WriteLine("timerSignals (reload events): " + FastSignalFactory.TimerSignals);
Console.WriteLine("PopulateAsync (fast provider): " + FastSignalFactory.PopulateCount);
Console.WriteLine("PopulateAsync (slow provider): " + SlowNoopFactory.PopulateCount);
Console.WriteLine();
Console.WriteLine(
"If timerSignals is much greater than PopulateAsync on the fast provider, " +
"change notifications were likely lost while refreshes were in flight.");

internal sealed class FastSignalFactory : IServiceEndpointProviderFactory
{
public static long TimerSignals;
public static long PopulateCount;

public bool TryCreateProvider(ServiceEndpointQuery query, [NotNullWhen(true)] out IServiceEndpointProvider? provider)
{
if (query.ServiceName.Equals("repro-service", StringComparison.OrdinalIgnoreCase))
{
provider = new FastProvider();
return true;
}

provider = null;
return false;
}
}

internal sealed class FastProvider : IServiceEndpointProvider
{
private readonly ReloadTokenSource _reload = new();
private CancellationTokenSource? _timerCts;
private Task? _timerTask;

public async ValueTask PopulateAsync(IServiceEndpointBuilder endpointBuilder, CancellationToken cancellationToken)
{
Interlocked.Increment(ref FastSignalFactory.PopulateCount);

if (_timerTask is null)
{
_timerCts = new CancellationTokenSource();
_timerTask = RunTimerAsync(_timerCts.Token);
}

endpointBuilder.Endpoints.Add(ServiceEndpoint.Create(new DnsEndPoint("fast", 80)));
endpointBuilder.AddChangeToken(_reload.GetChangeToken());
await Task.Delay(400, cancellationToken).ConfigureAwait(false);
}

private async Task RunTimerAsync(CancellationToken cancellationToken)
{
try
{
using var timer = new PeriodicTimer(TimeSpan.FromMilliseconds(120));
while (await timer.WaitForNextTickAsync(cancellationToken).ConfigureAwait(false))
{
Interlocked.Increment(ref FastSignalFactory.TimerSignals);
_reload.OnReload();
}
}
catch (OperationCanceledException)
{
}
}

public ValueTask DisposeAsync()
{
_timerCts?.Cancel();
_timerCts?.Dispose();
return ValueTask.CompletedTask;
}
}

internal sealed class SlowNoopFactory : IServiceEndpointProviderFactory
{
public static long PopulateCount;

public bool TryCreateProvider(ServiceEndpointQuery query, [NotNullWhen(true)] out IServiceEndpointProvider? provider)
{
if (query.ServiceName.Equals("repro-service", StringComparison.OrdinalIgnoreCase))
{
provider = new SlowProvider();
return true;
}

provider = null;
return false;
}
}

internal sealed class SlowProvider : IServiceEndpointProvider
{
public async ValueTask PopulateAsync(IServiceEndpointBuilder endpointBuilder, CancellationToken cancellationToken)
{
Interlocked.Increment(ref SlowNoopFactory.PopulateCount);
await Task.Delay(400, cancellationToken).ConfigureAwait(false);
endpointBuilder.Endpoints.Add(ServiceEndpoint.Create(new DnsEndPoint("slow", 81)));
}

public ValueTask DisposeAsync() => ValueTask.CompletedTask;
}

internal sealed class ReloadTokenSource
{
private CancellationTokenSource _cts = new();

public IChangeToken GetChangeToken() => new CancellationChangeToken(_cts.Token);

public void OnReload()
{
var newCts = new CancellationTokenSource();
CancellationTokenSource oldCts = Interlocked.Exchange(ref _cts, newCts);
oldCts.Cancel();
oldCts.Dispose();
}
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.