microsoft / microsoft/aspire

PostgreSQL 18 initdb-restart window: native AddDatabase creation races it, and the resource reports Healthy before the restart completes (WaitFor dependents released into a connection reset)

Open
#18,540 1 comment 0 reactions 0 assignees View on GitHub
area-integrations triage:bot-seen
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues (related but distinct: #11710 PG18+DataVolume, #11980 AddDatabase FATAL with WithCreationScript, #11337 stale-volume auth, #13712 WithInitFiles default-DB-only)

### Describe the bug

On the PostgreSQL 18 image, `initdb` runs the entrypoint scripts on the unix socket with TCP **closed**, then **restarts** the server to the real listener — the container emits two "database system is ready to accept connections" lines. Aspire's Postgres integration does not account for that restart window, which produces two distinct failures during a fresh first start:

**1. Native `AddDatabase` creation races the restart.**
Aspire creates the per-server databases once, on the server's `ResourceReadyEvent` (when the `postgres_check` `SELECT 1` passes), by opening a single connection to `postgres` and looping `CREATE DATABASE` over the declared databases. This is single-shot — no retry — and its catch swallows only `42P04 duplicate_database`. When the creation lands inside the initdb-restart window, the connection is reset mid-flight, or a concurrent creation attempt yields `23505 unique_violation` on `pg_database_datname_index`, which falls through to the generic catch and logs **`Failed to create database`**. There is no retry knob and no opt-out.

**2. The resource reports Healthy before the restart finishes, so `WaitFor` releases dependents into a connection reset.**
`postgres_check` passes on the **first** `SELECT 1`, which can happen **before** the initdb restart. The resource flips Healthy, every `WaitFor(db)` dependent is released, and they connect straight into the restart — the server accepts a TCP connection and then resets it mid-TLS-handshake. The dependent's startup connection throws and the resource goes **FailedToStart**. From the outside this looks like a flaky/slow boot; it is actually a premature Healthy signal.

### Expected behavior

- Native database creation should be resilient to the initdb-restart window — retry across a reset and treat an already-existing database (`23505` as well as `42P04`) as success, or expose a way to defer/disable it.
- The Postgres resource should not report Healthy (and therefore should not release `WaitFor` dependents) until the server is **durably** past the initdb restart — e.g. require a connection that survives N consecutive probes over a short window, rather than flipping Healthy on the first `SELECT 1`.

### Steps to reproduce

Fresh data volume (no existing PGDATA), PostgreSQL 18 image, several databases, and a project that `WaitFor`s one of them:

```csharp
var builder = DistributedApplication.CreateBuilder(args);

var pg = builder.AddPostgres("pg"); // PG18 image
var db1 = pg.AddDatabase("db1");
var db2 = pg.AddDatabase("db2");
var db3 = pg.AddDatabase("db3");

builder.AddProject("svc")
.WithReference(db1)
.WaitFor(db1);

builder.Build().Run();
```

Start on a fresh volume, repeatedly. Intermittently you get `Failed to create database` in the Postgres resource log and/or `svc` failing to start with a connection-reset / mid-handshake error, despite `db1` having briefly shown Healthy.

### Workaround (for reference)

We stopped creating databases over TCP entirely and create them in `initdb` instead, where they cannot race the restart (the scripts run before TCP is exposed):

- `postgres.WithContainerFiles("/docker-entrypoint-initdb.d", ...)` to drop a `CREATE DATABASE` script generated from the model. (We used `WithContainerFiles` rather than `WithInitFiles` because of #13712 — `WithInitFiles` is limited to the default DB name.)
- A custom health check ("startup stability gate") attached to each database resource that stays Unhealthy until a single connection survives 6 consecutive `SELECT 1` (~3s) — proof the initdb restart is over — so `WaitFor` dependents are not released into the reset. It only ever delays Healthy and self-releases within a bounded window so it can never deadlock the stack.

Both are workarounds for behavior we'd expect Aspire's Postgres integration to own now that it ships a PG18 image.

### Environment

- Aspire 13.4.x
- PostgreSQL 18 image (as selected by `Aspire.Hosting.PostgreSQL`)
- Run mode (local dev), fresh data volume

Contributor guide

Open the contributing guide

Research direction

Start at the PostgreSQL resource's ResourceReadyEvent database-creation path and the postgres_check health probe. Reproduce with a fresh PostgreSQL 18 volume, several AddDatabase calls, and a WaitFor dependent. Done means creation survives the restart window and dependents are not released until the server remains available through the restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, postgresql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.