block / block/buzz

Migration 26 (replica_heartbeat) isn't idempotent — in-doubt commit after Postgres failover permanently strands relay pod boot

Open
#3,785 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Environment

- Relay image: `ghcr.io/block/buzz:main` (digest `sha256:9de8aff1...`)
- sqlx `0.9.0` (pinned in `Cargo.lock`)
- Self-hosted CloudNativePG Postgres 17.4, 3 replicas, `BUZZ_AUTO_MIGRATE=true`

## What happened

During a rolling deploy that also restarted all 3 Postgres instances concurrently (unrelated resource-request change to the CNPG `Cluster`), one relay pod's boot-time migration run landed migration 26 (`migrations/0026_replica_heartbeat.sql`) in an inconsistent state:

- `replica_heartbeat` table: created, fully formed (PK `id smallint CHECK (id=1)`), with the expected single row (`id=1`).
- `_operator_global_tables`: has the matching catalog row for `replica_heartbeat`.
- `_sqlx_migrations`: **no row at all for version 26** (not even `success=false`).

So migration 26's DDL/DML fully committed, but the migration ledger insert — which `sqlx::migrate::Migrate::apply` wraps in the *same* transaction as the migration body — never landed durably. Every subsequent pod boot re-reads `_sqlx_migrations`, sees version 26 as not-yet-applied, and unconditionally re-runs the script, which now dies immediately on `CREATE TABLE replica_heartbeat` (relation already exists). This is permanent: it does not self-heal on retry, because the underlying condition (ledger row missing, table present) never changes.

We don't think this is a gap in sqlx's own transaction/advisory-lock handling — two replicas booting concurrently already serialize correctly via `pg_advisory_lock`. The likely trigger is narrower: a relay pod committed migration 26's transaction right as the Postgres primary was mid-failover for an unrelated reason, and the commit's acknowledgment was lost to the client on the connection that got cut over, even though the transaction itself landed on the (new) primary. Rare, but not impossible in any self-hosted, HA CNPG/Patroni-style deployment where primary failover and relay restarts can overlap.

## Immediate workaround (what we did)

Since the DB is the source of truth here, we reverted exactly the orphaned state and let a fresh boot re-run the real migrator cleanly:

```sql
DROP TABLE replica_heartbeat;
DELETE FROM _operator_global_tables WHERE table_name = 'replica_heartbeat';
```

Confirmed: the next pod restart ran migration 26 (and the ledger insert) successfully, no manual `_sqlx_migrations` row forging needed.

## Proposed fix

`0026_replica_heartbeat.sql`'s three statements aren't idempotent (`CREATE TABLE`, `INSERT`, and the `_operator_global_tables` catalog insert are all unconditional). As defense-in-depth against exactly this in-doubt-commit-after-failover case — which sqlx's transaction/advisory-lock can't fully protect against, since the loss is on the *client's visibility of the commit ack*, not the transaction itself — we'd suggest:

```sql
CREATE TABLE IF NOT EXISTS replica_heartbeat (...);
INSERT INTO replica_heartbeat (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
-- and the equivalent ON CONFLICT DO NOTHING for the _operator_global_tables catalog row
```

More generally, any future migration that creates fixed/singleton state (not incremental per-tenant data) is a candidate for the same class of bug — idempotent DDL/DML there is cheap insurance against a rare but real failure mode in HA self-hosted deployments.

Possibly related to (but distinct from) #2472, which covers a different way `_sqlx_migrations` checksums can strand a deployment — flagging in case there's appetite for a broader look at migration robustness under HA/failover conditions.

## Impact

No data loss or corruption — the relay pod that hit this crash-loops indefinitely until manually unblocked (as above), but doesn't affect already-running replicas. On a rolling deploy with `maxUnavailable=0`, this manifests as the deploy getting permanently stuck rather than an outage.

Contributor guide

Open the contributing guide

Research direction

Start with migrations/0026_replica_heartbeat.sql and the boot-time sqlx::migrate::Migrate::apply path described in the report. Exercise both a fresh database and the reported state where the table and catalog row exist but _sqlx_migrations lacks version 26. Done means a retry completes without crashing and preserves the expected singleton row and catalog state.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.