PostgreSQL 18 initdb-restart window: native AddDatabase creation races it, and the resource reports Healthy before the restart completes (WaitFor dependents released into a connection reset)
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues (related but distinct: #11710 PG18+DataVolume, #11980 AddDatabase FATAL with WithCreationScript, #11337 stale-volume auth, #13712 WithInitFiles default-DB-only)
### Describe the bug
On the PostgreSQL 18 image, `initdb` runs the entrypoint scripts on the unix socket with TCP **closed**, then **restarts** the server to the real listener — the container emits two "database system is ready to accept connections" lines. Aspire's Postgres integration does not account for that restart window, which produces two distinct failures during a fresh first start:
**1. Native `AddDatabase` creation races the restart.**
Aspire creates the per-server databases once, on the server's `ResourceReadyEvent` (when the `postgres_check` `SELECT 1` passes), by opening a single connection to `postgres` and looping `CREATE DATABASE` over the declared databases. This is single-shot — no retry — and its catch swallows only `42P04 duplicate_database`. When the creation lands inside the initdb-restart window, the connection is reset mid-flight, or a concurrent creation attempt yields `23505 unique_violation` on `pg_database_datname_index`, which falls through to the generic catch and logs **`Failed to create database`**. There is no retry knob and no opt-out.
**2. The resource reports Healthy before the restart finishes, so `WaitFor` releases dependents into a connection reset.**
`postgres_check` passes on the **first** `SELECT 1`, which can happen **before** the initdb restart. The resource flips Healthy, every `WaitFor(db)` dependent is released, and they connect straight into the restart — the server accepts a TCP connection and then resets it mid-TLS-handshake. The dependent's startup connection throws and the resource goes **FailedToStart**. From the outside this looks like a flaky/slow boot; it is actually a premature Healthy signal.
### Expected behavior
- Native database creation should be resilient to the initdb-restart window — retry across a reset and treat an already-existing database (`23505` as well as `42P04`) as success, or expose a way to defer/disable it.
- The Postgres resource should not report Healthy (and therefore should not release `WaitFor` dependents) until the server is **durably** past the initdb restart — e.g. require a connection that survives N consecutive probes over a short window, rather than flipping Healthy on the first `SELECT 1`.
### Steps to reproduce
Fresh data volume (no existing PGDATA), PostgreSQL 18 image, several databases, and a project that `WaitFor`s one of them:
```csharp
var builder = DistributedApplication.CreateBuilder(args);
var pg = builder.AddPostgres("pg"); // PG18 image
var db1 = pg.AddDatabase("db1");
var db2 = pg.AddDatabase("db2");
var db3 = pg.AddDatabase("db3");
builder.AddProject("svc")
.WithReference(db1)
.WaitFor(db1);
builder.Build().Run();
```
Start on a fresh volume, repeatedly. Intermittently you get `Failed to create database` in the Postgres resource log and/or `svc` failing to start with a connection-reset / mid-handshake error, despite `db1` having briefly shown Healthy.
### Workaround (for reference)
We stopped creating databases over TCP entirely and create them in `initdb` instead, where they cannot race the restart (the scripts run before TCP is exposed):
- `postgres.WithContainerFiles("/docker-entrypoint-initdb.d", ...)` to drop a `CREATE DATABASE` script generated from the model. (We used `WithContainerFiles` rather than `WithInitFiles` because of #13712 — `WithInitFiles` is limited to the default DB name.)
- A custom health check ("startup stability gate") attached to each database resource that stays Unhealthy until a single connection survives 6 consecutive `SELECT 1` (~3s) — proof the initdb restart is over — so `WaitFor` dependents are not released into the reset. It only ever delays Healthy and self-releases within a bounded window so it can never deadlock the stack.
Both are workarounds for behavior we'd expect Aspire's Postgres integration to own now that it ships a PG18 image.
### Environment
- Aspire 13.4.x
- PostgreSQL 18 image (as selected by `Aspire.Hosting.PostgreSQL`)
- Run mode (local dev), fresh data volume
Contributor guide
Research direction
Start at the PostgreSQL resource's ResourceReadyEvent database-creation path and the postgres_check health probe. Reproduce with a fresh PostgreSQL 18 volume, several AddDatabase calls, and a WaitFor dependent. Done means creation survives the restart window and dependents are not released until the server remains available through the restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, postgresql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100