block / block/buzz

Migration files edited in place break sqlx checksums for any long-lived database on image upgrade

Open
#2,472 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Environment

- Database migrated by `ghcr.io/block/buzz:sha-2d26db6`
(`2d26db6…`, 2026-06-16, includes auto-migrate #988)
- Upgraded to `ghcr.io/block/buzz:sha-acfbb1b`
(`acfbb1bb…`, 2026-07-22 main)
- External Cloud SQL Postgres 16, `BUZZ_AUTO_MIGRATE=true`

## What happens

The newer relay refuses to start against the June-migrated database:

```
ERROR Failed to run database migrations: migration error: migration 1 was previously applied but has been modified
```

sqlx records a checksum per applied migration; at least migration 1 was
edited in place between 2026-06-16 and 2026-07-22, so every database migrated
in that window fails checksum validation on upgrade. There is no forward path
for the deployment at that point — the options are hand-editing
`_sqlx_migrations` rows (leaves schema drift if the edit changed shape) or
dropping the schema (we did the latter; it was a pilot with test data — a
production tenant DB could not do this).

## Why this matters for deployers

Buzz positions the relay as self-hostable with `BUZZ_AUTO_MIGRATE` as the
upgrade path (#988). In-place edits to already-shipped migrations silently
convert every existing database into an upgrade dead-end. Anyone running a
relay across that window — self-hosters included — hits this on their next
image pull.

## Proposal

1. **Append-only discipline for shipped migrations** — once a migration has
appeared in any published image (ghcr tags count), it is frozen; changes
land as new migrations.
2. If a squash/reset is ever genuinely needed, ship it as an explicit,
versioned **baseline migration with a documented cut-over** (e.g. "relay
≥ X requires DB baseline Y; run the provided re-baseline tool"), not as a
silent edit.
3. A CI check on the repo enforcing (1) is cheap: compare checksums of
`migrations/*` against the previous release tag and fail on modification.

Happy to contribute the CI check.

Contributor guide

Open the contributing guide

Research direction

Start by locating migrations/* and the repository's release or CI configuration; review how sqlx checksums applied migrations in _sqlx_migrations and compare files with the previous release tag. Done means the check detects edits to migrations present in a published image, permits new migrations, and clearly reports the failing migration.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
ci-cd, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.