Service with multiple replicas is never reported healthy unless all replicas are healthy
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Describe the bug
An Aspire .NET project consisting of multiple replicas never becomes healthy unless _all_ replica instances have become healthy. As a result, dependent apps wait forever.
### Expected Behavior
When using multiple replicas, I would expect the service to become healthy when _at least one_ instance is healthy, so that dependent apps that rely on load-balancing can start.
### Steps To Reproduce
1. Create a new Aspire project from the starter template and configure multiple replicas for `apiservice` in the apphost:
```c#
var service = builder.AddProject...("apiservice")
// ... existing code
.WithReplicas(3);
```
2. In the ApiService project, add the following to `Program.cs`:
```c#
using var processLock = new CrossProcessLock();
// ... existing code
if (!processLock.IsFirstInstance())
{
builder.Services.AddHealthChecks()
.AddCheck("always-unhealthy", () => HealthCheckResult.Unhealthy());
}
WebApplication app = builder.Build();
// ... existing code
app.Run();
processLock.KeepAlive();
internal sealed class CrossProcessLock : IDisposable
{
private readonly Mutex _mutex;
public CrossProcessLock()
{
_mutex = new Mutex(initiallyOwned: false, "MyUniqueAppNameLock");
}
public bool IsFirstInstance()
{
try
{
if (_mutex.WaitOne(TimeSpan.FromSeconds(1), exitContext: false))
{
return true;
}
}
catch (OperationCanceledException)
{
}
return false;
}
[MethodImpl(MethodImplOptions.NoInlining)]
public void KeepAlive()
{
}
public void Dispose()
{
_mutex.ReleaseMutex();
_mutex.Dispose();
}
}
```
3. Run the app and observe on the dashboard that:
- All three instances remain unhealthy
- The first instance is reported as unhealthy repeatedly
- The dependent app never completes startup
### Exceptions (if any)
None
### .NET Version info
.NET SDK:
Version: 10.0.102
Commit: 4452502459
Workload version: 10.0.100-manifests.6d969a7e
MSBuild version: 18.0.7+445250245
Runtime Environment:
OS Name: Windows
OS Version: 10.0.26100
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\10.0.102\
.NET workloads installed:
There are no installed workloads to display.
Configured to use workload sets when installing new manifests.
No workload sets are installed. Run "dotnet workload restore" to install a workload set.
Host:
Version: 10.0.2
Architecture: x64
Commit: 4452502459
.NET SDKs installed:
10.0.101 [C:\Program Files\dotnet\sdk]
10.0.102 [C:\Program Files\dotnet\sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.23 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 10.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 10.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 8.0.23 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 10.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 10.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 8.0.23 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 10.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 10.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]
Environment variables:
Not set
global.json file:
Not found
### Anything else?
_No response_
Contributor guide
Research direction
Reproduce the issue from the starter template by configuring three replicas for apiservice in the apphost and adding the health check code in Program.cs. Observe how replica health affects the dashboard and dependent-app startup. Done means the service is reported healthy and dependent apps can start when at least one replica is healthy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100