microsoft / microsoft/aspire

.WaitFor() not working for databases with Docker Compose Publisher

Open
#9,194 6 comments 2 reactions 0 assignees View on GitHub
area-deployment docker-compose silent-failure
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

I was trying to setup a Database migration service using EF migrations when I noticed, that my migration service starts too early and fails because Postgres is not ready yet. When launching my aspire app out of Visual Studio it works flawlessly, however once I try to fire up the docker-compose.yaml generated by aspire publish and the new DockerComposePublisher using Podman as my docker environment It no longer does.

The reason why I consider this a timing error is because if I add an await Task.Delay(TimeSpan.FromSeconds(10)) into my migration service that waits a bit before starting the migrations they execute flawlessly.

### Expected Behavior

Identical WaitFor behavior between Aspire publish and Visual Studio launch

### Steps To Reproduce

The relevant part of the AppHost code I am using looks like this:
```csharp
var postgres = builder.AddPostgres("postgres")
.WithLifetime(ContainerLifetime.Session);

var machineDb = postgres.AddDatabase("machineStateDb");

var migrationService = builder.AddProject("migration")
.WithReference(machineDb)
.WaitFor(machineDb);

builder.AddDockerComposePublisher();
builder.Build().Run();
```

Then run aspire publish and select the docker-compose publisher and launch the compose using "podman compose up"

Any Client to the database that tries to connect to it right on startup should work.

### Exceptions (if any)

```
postgres-1 | The files belonging to this database system will be owned by user "postgres".
postgres-1 | This user must also own the server process.
postgres-1 |
postgres-1 | The database cluster will be initialized with locale "en_US.utf8".
postgres-1 | The default database encoding has accordingly been set to "UTF8".
postgres-1 | The default text search configuration will be set to "english".
postgres-1 |
postgres-1 | Data page checksums are disabled.
postgres-1 |
postgres-1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres-1 | creating subdirectories ... ok
postgres-1 | selecting dynamic shared memory implementation ... posix
postgres-1 | selecting default "max_connections" ... 100
postgres-1 | selecting default "shared_buffers" ... 128MB
postgres-1 | selecting default time zone ... Etc/UTC
postgres-1 | creating configuration files ... ok
postgres-1 | running bootstrap script ... ok
postgres-1 | performing post-bootstrap initialization ... ok
postgres-1 | syncing data to disk ... ok
postgres-1 |
postgres-1 | Success. You can now start the database server using:
postgres-1 |
postgres-1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres-1 |
postgres-1 | waiting for server to start....2025-05-09 06:52:27.743 UTC [43] LOG: starting PostgreSQL 17.2 (Debian 17.2-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgres-1 | 2025-05-09 06:52:27.745 UTC [43] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres-1 | 2025-05-09 06:52:27.749 UTC [46] LOG: database system was shut down at 2025-05-09 06:52:27 UTC
postgres-1 | 2025-05-09 06:52:27.752 UTC [43] LOG: database system is ready to accept connections
postgres-1 | done
postgres-1 | server started
postgres-1 | CREATE DATABASE
postgres-1 |
postgres-1 |
migration-1 | info: Microsoft.Hosting.Lifetime[0]
postgres-1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
migration-1 | Application started. Press Ctrl+C to shut down.
postgres-1 |
migration-1 | info: Microsoft.Hosting.Lifetime[0]
postgres-1 | 2025-05-09 06:52:27.912 UTC [43] LOG: received fast shutdown request
migration-1 | Hosting environment: Production
postgres-1 | waiting for server to shut down....2025-05-09 06:52:27.913 UTC [43] LOG: aborting any active transactions
migration-1 | info: Microsoft.Hosting.Lifetime[0]
postgres-1 | 2025-05-09 06:52:27.914 UTC [43] LOG: background worker "logical replication launcher" (PID 49) exited with exit code 1
migration-1 | Content root path: /app
postgres-1 | 2025-05-09 06:52:27.914 UTC [44] LOG: shutting down
migration-1 | fail: Microsoft.EntityFrameworkCore.Database.Connection[20004]
postgres-1 | 2025-05-09 06:52:27.916 UTC [44] LOG: checkpoint starting: shutdown immediate
migration-1 | An error occurred using the connection to database 'machineStateDb' on server 'tcp://postgres:5432'.
postgres-1 | 2025-05-09 06:52:27.961 UTC [44] LOG: checkpoint complete: wrote 921 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.009 s, sync=0.033 s, total=0.047 s; sync files=301, longest=0.002 s, average=0.001 s; distance=4238 kB, estimate=4238 kB; lsn=0/1908970, redo lsn=0/1908970
migration-1 | fail: Microsoft.Extensions.Hosting.Internal.Host[9]
postgres-1 | 2025-05-09 06:52:27.965 UTC [43] LOG: database system is shut down
migration-1 | BackgroundService failed
migration-1 | System.InvalidOperationException: An exception has been raised that is likely due to a transient failure.
migration-1 | ---> Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.89.1.16:5432
migration-1 | ---> System.Net.Sockets.SocketException (111): Connection refused
migration-1 | at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
migration-1 | at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
migration-1 | at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
migration-1 | at Npgsql.TaskTimeoutAndCancellation.ExecuteAsync(Func`2 getTaskFunc, NpgsqlTimeout timeout, CancellationToken cancellationToken)
migration-1 | at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
postgres-1 | done
migration-1 | at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
postgres-1 | server stopped
migration-1 | at Npgsql.Internal.NpgsqlConnector.RawOpen(SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
postgres-1 |
migration-1 | at Npgsql.Internal.NpgsqlConnector.g__OpenCore|214_1(NpgsqlConnector conn, SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
postgres-1 | PostgreSQL init process complete; ready for start up.
migration-1 | at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
postgres-1 |
migration-1 | at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
postgres-1 | 2025-05-09 06:52:28.027 UTC [1] LOG: starting PostgreSQL 17.2 (Debian 17.2-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
migration-1 | at Npgsql.PoolingDataSource.g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
migration-1 | at Npgsql.NpgsqlConnection.g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
postgres-1 | 2025-05-09 06:52:28.054 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
migration-1 | at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
postgres-1 | 2025-05-09 06:52:28.054 UTC [1] LOG: listening on IPv6 address "::", port 5432
migration-1 | at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
postgres-1 | 2025-05-09 06:52:28.056 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
migration-1 | at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
postgres-1 | 2025-05-09 06:52:28.059 UTC [59] LOG: database system was shut down at 2025-05-09 06:52:27 UTC
migration-1 | at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
postgres-1 | 2025-05-09 06:52:28.063 UTC [1] LOG: database system is ready to accept connections
migration-1 | at Microsoft.EntityFrameworkCore.Migrations.HistoryRepository.GetAppliedMigrationsAsync(CancellationToken cancellationToken)
migration-1 | at Npgsql.EntityFrameworkCore.PostgreSQL.Migrations.Internal.NpgsqlHistoryRepository.GetAppliedMigrationsAsync(CancellationToken cancellationToken)
migration-1 | at Npgsql.EntityFrameworkCore.PostgreSQL.Migrations.Internal.NpgsqlMigrator.MigrateAsync(String targetMigration, CancellationToken cancellationToken)
migration-1 | at DatabaseMigrationService.ApiDbInitializer.<>c__DisplayClass7_0.<b__0>d.MoveNext() in C:\Repos\Datron.Jaguar.AspireDashboard\Model\DatabaseMigrationService\DbInitializer.cs:line 68
migration-1 | --- End of stack trace from previous location ---
migration-1 | at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.<>c.<b__3_0>d.MoveNext()
migration-1 | --- End of stack trace from previous location ---
migration-1 | at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
migration-1 | --- End of inner exception stack trace ---
migration-1 | at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
migration-1 | at DatabaseMigrationService.ApiDbInitializer.RunMigrationAsync(WeatherContext dbContext, CancellationToken cancellationToken) in C:\Repos\Datron.Jaguar.AspireDashboard\Model\DatabaseMigrationService\DbInitializer.cs:line 62
migration-1 | at DatabaseMigrationService.ApiDbInitializer.ExecuteAsync(CancellationToken cancellationToken) in C:\Repos\Datron.Jaguar.AspireDashboard\Model\DatabaseMigrationService\DbInitializer.cs:line 32
migration-1 | at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
migration-1 | crit: Microsoft.Extensions.Hosting.Internal.Host[10]
migration-1 | The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
migration-1 | System.InvalidOperationException: An exception has been raised that is likely due to a transient failure.
migration-1 | ---> Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.89.1.16:5432
migration-1 | ---> System.Net.Sockets.SocketException (111): Connection refused
migration-1 | at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
migration-1 | at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
migration-1 | at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
migration-1 | at Npgsql.TaskTimeoutAndCancellation.ExecuteAsync(Func`2 getTaskFunc, NpgsqlTimeout timeout, CancellationToken cancellationToken)
migration-1 | at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
migration-1 | at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
migration-1 | at Npgsql.Internal.NpgsqlConnector.RawOpen(SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
migration-1 | at Npgsql.Internal.NpgsqlConnector.g__OpenCore|214_1(NpgsqlConnector conn, SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
migration-1 | at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
migration-1 | at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
migration-1 | at Npgsql.PoolingDataSource.g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
migration-1 | at Npgsql.NpgsqlConnection.g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
migration-1 | at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
migration-1 | at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
migration-1 | at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
migration-1 | at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
migration-1 | at Microsoft.EntityFrameworkCore.Migrations.HistoryRepository.GetAppliedMigrationsAsync(CancellationToken cancellationToken)
migration-1 | at Npgsql.EntityFrameworkCore.PostgreSQL.Migrations.Internal.NpgsqlHistoryRepository.GetAppliedMigrationsAsync(CancellationToken cancellationToken)
migration-1 | at Npgsql.EntityFrameworkCore.PostgreSQL.Migrations.Internal.NpgsqlMigrator.MigrateAsync(String targetMigration, CancellationToken cancellationToken)
migration-1 | at DatabaseMigrationService.ApiDbInitializer.<>c__DisplayClass7_0.<b__0>d.MoveNext() in C:\Repos\Datron.Jaguar.AspireDashboard\Model\DatabaseMigrationService\DbInitializer.cs:line 68
migration-1 | --- End of stack trace from previous location ---
migration-1 | at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.<>c.<b__3_0>d.MoveNext()
migration-1 | --- End of stack trace from previous location ---
migration-1 | at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
migration-1 | --- End of inner exception stack trace ---
migration-1 | at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
migration-1 | at DatabaseMigrationService.ApiDbInitializer.RunMigrationAsync(WeatherContext dbContext, CancellationToken cancellationToken) in C:\Repos\Datron.Jaguar.AspireDashboard\Model\DatabaseMigrationService\DbInitializer.cs:line 62
migration-1 | at DatabaseMigrationService.ApiDbInitializer.ExecuteAsync(CancellationToken cancellationToken) in C:\Repos\Datron.Jaguar.AspireDashboard\Model\DatabaseMigrationService\DbInitializer.cs:line 32
migration-1 | at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
migration-1 | info: Microsoft.Hosting.Lifetime[0]
migration-1 | Application is shutting down...
```

### .NET Version info


### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.