microsoft / microsoft/aspire

Concurrency issue with password generation of persistent containers

Open
#13,720 1 comment 0 reactions 0 assignees View on GitHub
area-app-model
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

If you have a persistent resource with an auto generated password, and run multiple, `DistributedApplicationTestingBuilder` tests against the resoruce concurrently, some of the instances may fail to go healthy, complaining `WRONGPASS invalid username-password pair or user is disabled.`

I believe there's some kind of race condition with both tests trying to write the user secret, and then disagreeing on what the value should actually be. But since they both share the same persistent container, only one of them will actually be right with the secret value.

I've only repro'd this with Redis, but I assume it may be a more general issue with password generated parameters, rather than redis specifically.

### Expected Behavior

When running tests, both tests should go healthy, without the `WRONGPASS invalid username-password pair or user is disabled.` health check failure.

If you compare the value in user secrets, to the `REDIS_PASSWORD` env var in docker desktop, you can see they are different - I'd expect these to be the same

Image

### Steps To Reproduce

Repro Steps:

```bash
# Clear any running persistent containers
docker rm $(docker ps -a -q) -f

# Reset user secret
dotnet user-secrets --project ..\..\src\AppHost\ remove 'Parameters:cache-password'

# Run tests
dotnet test --output detailed
```

Note, it's important to ensure you're starting from a blank slate (no user secrets, no running containers) to reproduce this issue. Which reproduces a new dev experience, and CI environments.

Tests (xunit):
```cs
public abstract class IntegrationTestBase
{
[Fact]
public async Task WeatherResponds()
{
var cts = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken);
cts.CancelAfter(TimeSpan.FromSeconds(60));

await using var appHost = await DistributedApplicationTestingBuilder.CreateAsync(cts.Token);
// Work around https://github.com/dotnet/aspire/issues/13714
appHost.Services.AddLogging(logging =>
{
logging.SetMinimumLevel(LogLevel.Debug)
.AddFilter("Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService", LogLevel.Information);
});

await using var app = await appHost.BuildAsync(cts.Token);

await app.StartAsync(cts.Token);

// Act
await app.ResourceNotifications.WaitForResourceHealthyAsync("webfrontend", cts.Token);

using var httpClient = app.CreateHttpClient("webfrontend");
using var response = await httpClient.GetAsync("/weather", cts.Token);

// Assert
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
}

// Need the test in multiple classes for xunit to parallelise the test.
public class IntegrationTests1 : IntegrationTestBase { }
public class IntegrationTests2 : IntegrationTestBase { }
```

App Host (based on default sample app)
```cs
using Microsoft.Extensions.DependencyInjection;

var builder = DistributedApplication.CreateBuilder(args);

var cache = builder.AddRedis("cache")
.WithLifetime(ContainerLifetime.Persistent)
.WithOtlpExporter();

var apiService = builder.AddProject("apiservice")
.WithHttpHealthCheck("/health");

builder.AddProject("webfrontend")
.WithExternalHttpEndpoints()
//.WithHttpHealthCheck("/health")
.WithReference(cache)
.WaitFor(cache)
.WithReference(apiService)
.WaitFor(apiService);

builder.Build().Run();
```

### Exceptions (if any)

```
fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
Health check cache_check with status Unhealthy completed after 11.3885ms with message '(null)'
StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). There was an authentication failure; check that passwords (or client certificates) are configured correctly: (RedisServerException) Error: WRONGPASS invalid username-password pair or user is disabled.
---> StackExchange.Redis.RedisServerException: Error: WRONGPASS invalid username-password pair or user is disabled.
--- End of inner exception stack trace ---
at StackExchange.Redis.ConnectionMultiplexer.ConnectImplAsync(ConfigurationOptions configuration, TextWriter writer, Nullable`1 serverType) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 596
at HealthChecks.Redis.RedisHealthCheck.TimeoutAsync(Task`1 task, CancellationToken cancellationToken) in /home/runner/work/AspNetCore.Diagnostics.HealthChecks/AspNetCore.Diagnostics.HealthChecks/src/HealthChecks.Redis/RedisHealthCheck.cs:line 124
at HealthChecks.Redis.RedisHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken) in /home/runner/work/AspNetCore.Diagnostics.HealthChecks/AspNetCore.Diagnostics.HealthChecks/src/HealthChecks.Redis/RedisHealthCheck.cs:line 54

```

### .NET Version info

_No response_

### Anything else?

For a fuller example, I first noticed this at https://github.com/afscrome/aspire-otel-testing/commit/dc53e837396b4f562b0280e30163b78eb3c66e6c, and fixing the issue required setting `--max-parallel-test-modules 1` on the `dotnet test` command - https://github.com/afscrome/aspire-otel-testing/commit/7b233444f5ab0f94d186c9c111cbc5688d23574a.

That scenario is slightly different in that the concurrency issue was happening across multiple test assemblies, rather than being parallelism within the same assembly, but I assume they're the same root cause.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.