microsoft / microsoft/aspire

Flakey tests if running Tests `WithEndpointProxySupport(false)` in parallel

Open
#10,146 2 comments 0 reactions 0 assignees View on GitHub
needs-retriage
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

If you have a persistent container, and run `DistributedApplicationTestingBuilder` tests in parallel, the two instances race to create the persistent containers and end up in a very confused state.

### Expected Behavior

Aspire shouldn't try to provision the persistent container multiple times.

### Steps To Reproduce

Run any instance of the below test by itself, the test will pass. Run both together, and the test hangs.
```cs
#pragma warning disable ASPIREPROXYENDPOINTS001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
[assembly: Parallelizable(ParallelScope.All)]

public class WebTests
{
[TestCase(1)]
[TestCase(2)]
public async Task Test(int instance)
{
var cts = new CancellationTokenSource(TimeSpan.FromSeconds(60));

var builder = DistributedApplicationTestingBuilder.Create();

var sql = builder.AddSqlServer("sql")
.WithHostPort(12345)
.WithEndpointProxySupport(false)
.WithLifetime(ContainerLifetime.Persistent);

var app = builder.Build();

await app.StartAsync(cts.Token);
await app.ResourceNotifications.WaitForResourceHealthyAsync(sql.Resource.Name, cts.Token);
}
}
#pragma warning restore ASPIREPROXYENDPOINTS001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
```

### Exceptions (if any)

The output varies from run to run, but there's the example of both tests from one run

Test(1)
```
info: Aspire.Hosting.DistributedApplication[0]
Aspire version: 9.4.0-preview.1.25329.2+f7fde8eb1f7586952b5563d416200948a939c7af
info: Aspire.Hosting.DistributedApplication[0]
Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
Application host directory is: S:\aspire91\aspire91.Tests
info: Aspire.Hosting.Dcp.DcpHost[0]
Starting DCP with arguments: start-apiserver --monitor 12596 --detach --kubeconfig "C:\Users\alexanderc\AppData\Local\Temp\aspire.zjt4gy1n.1xb\kubeconfig"
info: Aspire.Hosting.Dcp.dcp.start-apiserver.api-server[0]
Starting API server...
info: Aspire.Hosting.Dcp.dcp.start-apiserver.api-server[0]
API server started {"Address": "127.0.0.1", "Port": 52420}
info: Aspire.Hosting.Dcp.dcp.start-apiserver.dcp-host[0]
Starting DCP controller host
info: Aspire.Hosting.Dcp.dcp.start-apiserver.dcp-host[0]
Started all services {"count": 1}
info: Aspire.Hosting.Dcp.dcpctrl[0]
starting controller manager
info: Aspire.Hosting.DistributedApplication[0]
Distributed application started. Press Ctrl+C to shut down.
info: Aspire.Hosting.Dcp.dcpctrl.ResourceCleanup[0]
could not remove some abandoned containers. {"Error": "must specify at least one container"}
info: Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler[0]
found existing Container, but calculated lifecycle key doesn't match {"Container": {"name":"sql-125ea825"}, "Reconciliation": 2, "ContainerID": "020e8b01ea4972217cd93e18bb05b659c8ca8b82354019ddd555a7fc72a84b63", "OldLifecycleKey": "5adde0626999cdfed980dbb5ad23b9b9", "NewLifecycleKey": "bdf4ae11c41abeb2f2b9770848706df4", "MountChanges": [], "PortChanges": [], "EnvChanges": [], "OtherChanges": []}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 3, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: Aspire.Hosting.Dcp.dcpctrl.ResourceCleanup[0]
could not harvest all abandoned container networks {"Error": "must specify at least one network"}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 4, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 5, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 6, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 7, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 8, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: aspire91.Tests.Resources.sql[0]
1: 2025-06-30T17:13:06.4380000Z 979db5b2fbe88728fda901fa76c261e4e4f0af9ea9494fab7836a65bb2a85741
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 9, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler[0]
Added new ContainerNetworkConnection {"Container": {"name":"sql-125ea825"}, "Reconciliation": 5, "Container": "979db5b2fbe88728fda901fa76c261e4e4f0af9ea9494fab7836a65bb2a85741", "ContainerNetworkConnection": "/sql-125ea825-default-aspire-network-5l391nvqac"}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
not all expected containers are connected to the network, retrying... {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 10, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 0}
info: aspire91.Tests.Resources.sql[0]
2: 2025-06-30T17:13:08.8215461Z SQL Server 2022 will run as non-root by default.
info: aspire91.Tests.Resources.sql[0]
3: 2025-06-30T17:13:08.8215967Z This container is running as user mssql.
info: aspire91.Tests.Resources.sql[0]
4: 2025-06-30T17:13:08.8216008Z To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
info: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
service /sql is now in state Ready {"ServiceName": {"name":"sql"}, "Reconciliation": 4}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 11, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 12, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 13, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: aspire91.Tests.Resources.sql[0]
5: 2025-06-30T17:13:13.8491161Z 2025-06-30 17:13:13.83 Server Setup step is copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf'.
info: aspire91.Tests.Resources.sql[0]
6: 2025-06-30T17:13:13.8591312Z
info: aspire91.Tests.Resources.sql[0]
7: 2025-06-30 17:13:13.85 Server Did not find an existing master data file /var/opt/mssql/data/master.mdf, copying the missing default master and other system database files. If you have moved the database location, but not moved the database files, startup may fail. To repair: shutdown SQL Server, move the master database to configured location, and restart.
info: aspire91.Tests.Resources.sql[0]
8: 2025-06-30T17:13:13.8606037Z

// SNIP

info: aspire91.Tests.Resources.sql[0]
502: 2025-06-30T17:13:16.9711016Z
info: aspire91.Tests.Resources.sql[0]
503: 2025-06-30 17:13:16.96 spid46s Recovery is complete. This is an informational message only. No user action is required.
info: aspire91.Tests.Resources.sql[0]
504: 2025-06-30T17:13:16.9835063Z
info: aspire91.Tests.Resources.sql[0]
505: 2025-06-30 17:13:16.97 spid55s The default language (LCID 0) has been set for engine and full-text services.
info: aspire91.Tests.Resources.sql[0]
506: 2025-06-30T17:13:17.2117433Z
info: aspire91.Tests.Resources.sql[0]
507: 2025-06-30 17:13:17.20 spid55s The tempdb database has 8 data file(s).
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 14, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 15, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 16, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 17, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
```

Test(2)
```
info: Aspire.Hosting.DistributedApplication[0]
Aspire version: 9.4.0-preview.1.25329.2+f7fde8eb1f7586952b5563d416200948a939c7af
info: Aspire.Hosting.DistributedApplication[0]
Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
Application host directory is: S:\aspire91\aspire91.Tests
info: Aspire.Hosting.Dcp.DcpHost[0]
Starting DCP with arguments: start-apiserver --monitor 12596 --detach --kubeconfig "C:\Users\alexanderc\AppData\Local\Temp\aspire.bqz4gv5m.ybw\kubeconfig"
info: Aspire.Hosting.Dcp.dcp.start-apiserver.api-server[0]
Starting API server...
info: Aspire.Hosting.Dcp.dcp.start-apiserver.api-server[0]
API server started {"Address": "127.0.0.1", "Port": 52422}
info: Aspire.Hosting.Dcp.dcp.start-apiserver.dcp-host[0]
Starting DCP controller host
info: Aspire.Hosting.Dcp.dcp.start-apiserver.dcp-host[0]
Started all services {"count": 1}
info: Aspire.Hosting.DistributedApplication[0]
Distributed application started. Press Ctrl+C to shut down.
info: Aspire.Hosting.Dcp.dcpctrl[0]
starting controller manager
info: Aspire.Hosting.Dcp.dcpctrl.ResourceCleanup[0]
could not remove some abandoned containers. {"Error": "must specify at least one container"}
info: Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler[0]
found existing Container, but calculated lifecycle key doesn't match {"Container": {"name":"sql-125ea825"}, "Reconciliation": 2, "ContainerID": "020e8b01ea4972217cd93e18bb05b659c8ca8b82354019ddd555a7fc72a84b63", "OldLifecycleKey": "5adde0626999cdfed980dbb5ad23b9b9", "NewLifecycleKey": "70d6d529ce199d536cbb6903cd7a3053", "MountChanges": [], "PortChanges": [], "EnvChanges": [], "OtherChanges": []}
info: Aspire.Hosting.Dcp.dcpctrl.ResourceCleanup[0]
could not harvest all abandoned container networks {"Error": "must specify at least one network"}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 3, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 4, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 5, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 6, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 7, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 8, "Network": "default-aspire-network-125ea825", "Expected": 0, "Found": 0}
info: aspire91.Tests.Resources.sql[0]
1: 2025-06-30T17:13:06.4110000Z Error response from daemon: Conflict. The container name "/sql-125ea825" is already in use by container "979db5b2fbe88728fda901fa76c261e4e4f0af9ea9494fab7836a65bb2a85741". You have to remove (or rename) that container to be able to reuse that name.
info: Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler[0]
Added new ContainerNetworkConnection {"Container": {"name":"sql-125ea825"}, "Reconciliation": 5, "Container": "979db5b2fbe88728fda901fa76c261e4e4f0af9ea9494fab7836a65bb2a85741", "ContainerNetworkConnection": "/sql-125ea825-default-aspire-network-vjb99of8eo"}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
not all expected containers are connected to the network, retrying... {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 9, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 0}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 10, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 11, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 12, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
service /sql is now in state Ready {"ServiceName": {"name":"sql"}, "Reconciliation": 4}
info: aspire91.Tests.Resources.sql[0]
2: 2025-06-30T17:13:08.8215461Z SQL Server 2022 will run as non-root by default.
info: aspire91.Tests.Resources.sql[0]
3: 2025-06-30T17:13:08.8215967Z This container is running as user mssql.
info: aspire91.Tests.Resources.sql[0]
4: 2025-06-30T17:13:08.8216008Z To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 13, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: aspire91.Tests.Resources.sql[0]
5: 2025-06-30T17:13:13.8491161Z 2025-06-30 17:13:13.83 Server Setup step is copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf'.
info: aspire91.Tests.Resources.sql[0]
6: 2025-06-30T17:13:13.8591312Z
info: aspire91.Tests.Resources.sql[0]
7: 2025-06-30 17:13:13.85 Server Did not find an existing master data file /var/opt/mssql/data/master.mdf, copying the missing default master and other system database files. If you have moved the database location, but not moved the database files, startup may fail. To repair: shutdown SQL Server, move the master database to configured location, and restart.

// SNIP

info: aspire91.Tests.Resources.sql[0]
503: 2025-06-30 17:13:16.96 spid46s Recovery is complete. This is an informational message only. No user action is required.
info: aspire91.Tests.Resources.sql[0]
504: 2025-06-30T17:13:16.9835063Z
info: aspire91.Tests.Resources.sql[0]
505: 2025-06-30 17:13:16.97 spid55s The default language (LCID 0) has been set for engine and full-text services.
info: aspire91.Tests.Resources.sql[0]
506: 2025-06-30T17:13:17.2117433Z
info: aspire91.Tests.Resources.sql[0]
507: 2025-06-30 17:13:17.20 spid55s The tempdb database has 8 data file(s).
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 14, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 15, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 16, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 17, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
info: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
all expected containers are connected to the network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 18, "Network": "default-aspire-network-125ea825", "Expected": 1, "Found": 1}
```

### .NET Version info

_No response_

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.