microsoft / microsoft/aspire

[13.5.1] Parallel tests are no longer stable

Open
#19,570 1 comment 0 reactions 0 assignees View on GitHub
area-app-model area-testing triage:bot-seen
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

After upgrading a project to aspire 13.5, I have a test project which has become incredibly flakey. If I runt he whole project, I'll get a large portion of my tests fail. But if I run them individually, they all pass fine.

This test suite passed fine in 13.4.6.

I'm still investigating further. So far what I've discovered is in the failing tests the resources fails to start becuase the container runtime goes unhealthy

> Resource mailhog/mailhog-jmdvbcmd changed state: RuntimeUnhealthy

That said, I think this is a symptom, not the root cause as I see these failures in some tests that only touch processes, no containers.

### Expected Behavior

Parallel tests that passed in 13.4.6 continue to pass in 13.5.

### Steps To Reproduce

Unknown right now. But here's an example of one of the failing tests

```cs
[Test]
[CancelAfter(300_000)]
public async Task GrafanaGoesHealthy(CancellationToken ct)
{
var builder = Common.CreateIntegrationAppHost();

var grafana = builder.AddGrafana("grafana");

using var app = builder.Build();
await app.StartAsync(ct);

await app.ResourceNotifications.WaitForResourceHealthyAsync(grafana.Resource.Name, ct);
}

internal static IDistributedApplicationTestingBuilder CreateIntegrationAppHost()
=> ConfigureIntegrationAppHost(DistributedApplicationTestingBuilder.Create([], (opt,_) =>
{
opt.DashboardApplicationName = TestContext.CurrentContext?.Test.Name;
}));

internal static IDistributedApplicationTestingBuilder CreateIntegrationAppHostWithDashboard()
{
var builder = DistributedApplicationTestingBuilder.Create([], (opt, _) =>
{
opt.DisableDashboard = false;
opt.DashboardApplicationName = TestContext.CurrentContext?.Test.Name;
});

builder.Configuration["ASPIRE_INTERACTIVITY_ENABLED"] = "false"; // Ensure Interaction service is disabled.

// This gets disabled when the dashboard is running, but without it, we won't see logs in CI
builder.Services.AddLogging(x => x.AddFilter("Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService", LogLevel.Information));

// When Dashboard is enabled, wait for will waite indefinitely if a dependency fails as the user could fix the dependency allowing the resource to start up
// This is undesirable in tests, so go back to failing fast
builder.Services.Configure(x => x.DefaultWaitBehavior = WaitBehavior.StopOnResourceUnavailable);

return ConfigureIntegrationAppHost(builder);
}

private static IDistributedApplicationTestingBuilder ConfigureIntegrationAppHost(IDistributedApplicationTestingBuilder builder)
{
builder
.WithTrayportDefaults()
.WithTrayportTestingDefaults()
.WithFinalStateLogging()
.WithTestAspireStore();

builder.Services.AddLogging(x => x
.AddFilter("Aspire.Hosting.ApplicationModel.ResourceNotificationService", LogLevel.Trace)
.SetMinimumLevel(LogLevel.Debug));

return builder;
}
```

### Exceptions (if any)

```
[14:42:30] info: Aspire.Hosting.DistributedApplication[0] Aspire AppHost version: 13.5.1+69db530a4816698cf1d5fa4557933e0ac4f127c6
[14:42:30] info: Aspire.Hosting.DistributedApplication[0] Distributed application starting.
[14:42:30] info: Aspire.Hosting.DistributedApplication[0] Application host directory is: S:\REDACTED\test\REDACTED.E2ETests
[14:42:48] info: Aspire.Hosting.Dcp.DcpHost[0] Starting DCP with arguments: start-apiserver --monitor 44996 --detach --kubeconfig "C:\Users\alexanderc\AppData\Local\Temp\aspire-dcpecmo5h1q.zzj\kubeconfig" --tls-cert-thumbprint "827C259F5FA6F23036837DBBDC200A758AB9F879"
[14:42:48] trce: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0] Version: 1 Resource mailhog/mailhog-jmdvbcmd update published: ResourceType = Container, CreationTimeStamp = (null), State = { Text = (null), Style = (null) }, IsHidden = False, HeathStatus = (null), ResourceReady = False, ExitCode = (null), Urls = { }, EnvironmentVariables = { }, Properties = { }, HealthReports = { mailhog_http_/images/hog.png_200_check = (null) }, Commands = { start (Start) = Disabled stop (Stop) = Hidden restart (Restart) = Disabled }
[14:42:49] info: REDACTED.E2ETests.Resources.mailhog[0] 1: 2026-08-21T13:42:30.0727749Z Rewriting container image to 'REDACTED'
[14:43:06] trce: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0] Version: 2 Resource mailhog/mailhog-jmdvbcmd update published: ResourceType = Container, CreationTimeStamp = (null), State = { Text = (null), Style = (null) }, IsHidden = False, HeathStatus = (null), ResourceReady = False, ExitCode = (null), Urls = { http = http://localhost:61057 smtp = smtp://localhost:61056 }, EnvironmentVariables = { }, Properties = { }, HealthReports = { mailhog_http_/images/hog.png_200_check = (null) }, Commands = { start (Start) = Disabled stop (Stop) = Hidden restart (Restart) = Disabled }
[14:43:06] trce: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0] Version: 3 Resource mailhog/mailhog-jmdvbcmd update published: ResourceType = Container, CreationTimeStamp = (null), State = { Text = (null), Style = (null) }, IsHidden = False, HeathStatus = (null), ResourceReady = False, ExitCode = (null), Urls = { smtp = smtp://localhost:61056 http = http://localhost:61057 }, EnvironmentVariables = { }, Properties = { container.image = REDACTED:v1.0.1 container.id = (null) container.command = (null) container.args = container.ports = container.lifetime = Session resource.appArgs = (null) resource.appArgsSensitivity = (null) }, HealthReports = { mailhog_http_/images/hog.png_200_check = (null) }, Commands = { start (Start) = Disabled stop (Stop) = Hidden restart (Restart) = Disabled }
[14:43:06] trce: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0] Version: 4 Resource mailhog/mailhog-jmdvbcmd update published: ResourceType = Container, CreationTimeStamp = (null), State = { Text = (null), Style = (null) }, IsHidden = False, HeathStatus = (null), ResourceReady = False, ExitCode = (null), Urls = { smtp = smtp://localhost:61056 http = http://localhost:61057 }, EnvironmentVariables = { }, Properties = { container.image = REDACTED:v1.0.1 container.id = (null) container.command = (null) container.args = container.ports = container.lifetime = Session resource.appArgs = (null) resource.appArgsSensitivity = (null) resource.connectionString = smtp://localhost:61056 }, HealthReports = { mailhog_http_/images/hog.png_200_check = (null) }, Commands = { start (Start) = Disabled stop (Stop) = Hidden restart (Restart) = Disabled }
[14:43:06] dbug: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0] Resource mailhog/mailhog-jmdvbcmd changed state: Starting
[14:43:06] trce: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0] Version: 5 Resource mailhog/mailhog-jmdvbcmd update published: ResourceType = Container, CreationTimeStamp = (null), State = { Text = Starting, Style = (null) }, IsHidden = False, HeathStatus = (null), ResourceReady = False, ExitCode = (null), Urls = { smtp = smtp://localhost:61056 http = http://localhost:61057 }, EnvironmentVariables = { }, Properties = { container.image = REDACTED:v1.0.1 container.id = (null) container.command = (null) container.args = container.ports = container.lifetime = Session resource.appArgs = (null) resource.appArgsSensitivity = (null) resource.connectionString = smtp://localhost:61056 }, HealthReports = { mailhog_http_/images/hog.png_200_check = (null) }, Commands = { start (Start) = Disabled stop (Stop) = Hidden restart (Restart) = Disabled }
[14:43:07] info: REDACTED.E2ETests.Resources.mailhog[0] 2: 2026-08-21T13:43:07.0558158Z Resource 'mailhog' has a certificate trust scope of 'System'. Automatically including system root certificates in the trusted configuration.
[14:43:07] info: Aspire.Hosting.DistributedApplication[0] Distributed application started. Press Ctrl+C to shut down.
[14:43:07] dbug: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0] Waiting for resource 'mailhog' to enter the 'Healthy' state.
[14:43:07] trce: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0] Version: 6 Resource mailhog/mailhog-jmdvbcmd update published: ResourceType = Container, CreationTimeStamp = 2026-08-21T13:43:07, State = { Text = (null), Style = (null) }, IsHidden = False, HeathStatus = (null), ResourceReady = False, ExitCode = (null), Urls = { smtp = smtp://localhost:61056 http = http://localhost:61057 }, EnvironmentVariables = { SSL_CERT_DIR = /usr/lib/ssl/aspire/certs SSL_CERT_FILE = /usr/lib/ssl/aspire/cert.pem }, Properties = { container.image = REDACTED:v1.0.1 container.id = (null) container.command = (null) container.args = container.ports = 1025, 8025 container.lifetime = Session resource.appArgs = resource.appArgsSensitivity = resource.connectionString = smtp://localhost:61056 }, HealthReports = { mailhog_http_/images/hog.png_200_check = (null) }, Commands = { start (Start) = Disabled stop (Stop) = Hidden restart (Restart) = Disabled }
[14:43:07] trce: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0] Resource mailhog/mailhog-jmdvbcmd update skipped because the snapshot is unchanged.
[14:43:07] dbug: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0] Resource mailhog/mailhog-jmdvbcmd changed state: RuntimeUnhealthy
[14:43:07] trce: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0] Version: 7 Resource mailhog/mailhog-jmdvbcmd update published: ResourceType = Container, CreationTimeStamp = 2026-08-21T13:43:07, State = { Text = RuntimeUnhealthy, Style = (null) }, IsHidden = False, HeathStatus = (null), ResourceReady = False, ExitCode = (null), Urls = { smtp = smtp://localhost:61056 http = http://localhost:61057 }, EnvironmentVariables = { SSL_CERT_DIR = /usr/lib/ssl/aspire/certs SSL_CERT_FILE = /usr/lib/ssl/aspire/cert.pem }, Properties = { container.image = REDACTED:v1.0.1 container.id = (null) container.command = (null) container.args = container.ports = 1025, 8025 container.lifetime = Session resource.appArgs = resource.appArgsSensitivity = resource.connectionString = smtp://localhost:61056 }, HealthReports = { mailhog_http_/images/hog.png_200_check = (null) }, Commands = { start (Start) = Disabled stop (Stop) = Hidden restart (Restart) = Disabled }
[14:43:07] fail: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0] Stopped waiting for resource 'mailhog' to become healthy because it failed to start.
[14:43:07] info: REDACTED.Hosting.Testing.HostedLifecycleServices.FinalStateLoggerService[0] Resource: "mailhog", Type: "MailHogContainerResource", ExitCode: (null), Health: (null), State: ResourceStateSnapshot { Text = RuntimeUnhealthy, Style = } Reports: HealthReportSnapshot { Name = mailhog_http_/images/hog.png_200_check, Status = , Description = , ExceptionText = , LastRunAt = }
```

### Aspire doctor output

_No response_

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Research direction

Run the named GrafanaGoesHealthy test both alone and in parallel with the suite, comparing behavior between Aspire 13.4.6 and 13.5.1. Read Common.CreateIntegrationAppHost, ConfigureIntegrationAppHost, and the ResourceNotificationService logs first; done means parallel tests no longer fail intermittently or report RuntimeUnhealthy resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.