microsoft / microsoft/aspire

Service with multiple replicas is never reported healthy unless all replicas are healthy

Open
#14,091 0 comments 0 reactions 0 assignees View on GitHub
area-app-model
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

An Aspire .NET project consisting of multiple replicas never becomes healthy unless _all_ replica instances have become healthy. As a result, dependent apps wait forever.

### Expected Behavior

When using multiple replicas, I would expect the service to become healthy when _at least one_ instance is healthy, so that dependent apps that rely on load-balancing can start.

### Steps To Reproduce

1. Create a new Aspire project from the starter template and configure multiple replicas for `apiservice` in the apphost:
```c#
var service = builder.AddProject...("apiservice")
// ... existing code
.WithReplicas(3);
```
2. In the ApiService project, add the following to `Program.cs`:
```c#

using var processLock = new CrossProcessLock();

// ... existing code

if (!processLock.IsFirstInstance())
{
builder.Services.AddHealthChecks()
.AddCheck("always-unhealthy", () => HealthCheckResult.Unhealthy());
}

WebApplication app = builder.Build();

// ... existing code

app.Run();
processLock.KeepAlive();

internal sealed class CrossProcessLock : IDisposable
{
private readonly Mutex _mutex;

public CrossProcessLock()
{
_mutex = new Mutex(initiallyOwned: false, "MyUniqueAppNameLock");
}

public bool IsFirstInstance()
{
try
{
if (_mutex.WaitOne(TimeSpan.FromSeconds(1), exitContext: false))
{
return true;
}
}
catch (OperationCanceledException)
{
}

return false;
}

[MethodImpl(MethodImplOptions.NoInlining)]
public void KeepAlive()
{
}

public void Dispose()
{
_mutex.ReleaseMutex();
_mutex.Dispose();
}
}
```
3. Run the app and observe on the dashboard that:
- All three instances remain unhealthy
- The first instance is reported as unhealthy repeatedly
- The dependent app never completes startup

Image

### Exceptions (if any)

None

### .NET Version info

.NET SDK:
Version: 10.0.102
Commit: 4452502459
Workload version: 10.0.100-manifests.6d969a7e
MSBuild version: 18.0.7+445250245

Runtime Environment:
OS Name: Windows
OS Version: 10.0.26100
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\10.0.102\

.NET workloads installed:
There are no installed workloads to display.
Configured to use workload sets when installing new manifests.
No workload sets are installed. Run "dotnet workload restore" to install a workload set.

Host:
Version: 10.0.2
Architecture: x64
Commit: 4452502459

.NET SDKs installed:
10.0.101 [C:\Program Files\dotnet\sdk]
10.0.102 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.23 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 10.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 10.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 8.0.23 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 10.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 10.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 8.0.23 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 10.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 10.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
Not set

global.json file:
Not found

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue from the starter template by configuring three replicas for apiservice in the apphost and adding the health check code in Program.cs. Observe how replica health affects the dashboard and dependent-app startup. Done means the service is reported healthy and dependent apps can start when at least one replica is healthy.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.