drizzle-team / drizzle-team/drizzle-orm
drizzle-kit migrate silently skips pending migrations when MAX(created_at) in __drizzle_migrations exceeds the journal `when` of pending entries
- Dominant language
- TypeScript
- Stars
- 35.8k
- Forks
- 1.6k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 4
Description
### Summary
`drizzle-kit migrate` (via `PgDialect.migrate` in `drizzle-orm/pg-core/dialect.js`, used by every Postgres driver — `neon-serverless`, `neon-http`, `node-postgres`, `postgres-js`, `vercel-postgres`) decides whether a journal entry has already been applied by selecting the row with the highest `created_at` from `drizzle.__drizzle_migrations` and skipping any pending journal entry whose `folderMillis` (i.e. `_journal.json` `when`) is ≤ that value. The skip is silent — no log, no warning, no non-zero exit. The CLI still prints `[✓] migrations applied successfully!` and exits `0`.
This is a correctness bug in any case where a row in `__drizzle_migrations` has a `created_at` that doesn't come from drizzle-kit's own happy-path insert. That includes:
- The official "init from an existing schema" workflow.
- Any manual repair (e.g. recovering from `_journal.json` ↔ table drift via `INSERT INTO drizzle.__drizzle_migrations (hash, created_at) VALUES (…, extract(epoch from now())*1000);`, a pattern documented across multiple ops runbooks and Stack Overflow answers).
- Any project that bootstraps `__drizzle_migrations` from an existing production schema before adopting drizzle-kit.
In all of these, `created_at` is wall-clock-derived rather than journal-derived, and the next `drizzle-kit migrate` invocation will silent-skip every pending migration whose `_journal.json` `when` is earlier than that wall-clock value — including migrations the operator has just generated and very much intends to apply.
### Source
`drizzle-orm/src/pg-core/dialect.ts` (compiled to `drizzle-orm/pg-core/dialect.js:56-71` in published builds), `PgDialect.migrate`. The relevant predicate is roughly:
```js
const lastDbMigration = await tx.execute<{ id: number; hash: string; created_at: string }>(
sql`select id, hash, created_at from ${migrationsTable} order by created_at desc limit 1`,
);
// ...
if (!lastDbMigration[0] || Number(lastDbMigration[0].created_at) < migration.folderMillis) {
// apply migration
}
```
The single-row `MAX(created_at)` watermark is the root cause: a single poisoned `created_at` masks every later journal entry whose `when` ≤ that value, even if no row with that entry's `hash` exists in the table.
### Repro (any Postgres-backed app using `drizzle-kit migrate`)
1. Bootstrap a fresh Postgres database (any of the supported drivers).
2. Generate two drizzle migrations a few minutes apart so `_journal.json` has two entries with distinct `when` values, e.g. `1700000000000` and `1700000060000`.
3. Apply only the first migration via `drizzle-kit migrate` (or `psql` + `INSERT INTO drizzle.__drizzle_migrations (hash, created_at) VALUES ('', 1700000000000);`). Verify the table has one row with `created_at = 1700000000000`.
4. Now insert a poisoned-watermark row to simulate the manual-repair / seed-from-existing pattern: `INSERT INTO drizzle.__drizzle_migrations (hash, created_at) VALUES ('manual-repair-marker', 9999999999999);` (or use `extract(epoch from now())*1000` for any wall-clock value > the second journal entry's `when`).
5. Generate a third migration, e.g. `_journal.json` `when = 1700000120000`.
6. Run `drizzle-kit migrate`.
**Expected:** the third migration applies, OR drizzle-kit errors out citing the poisoned row.
**Actual:** the third migration is silently skipped. CLI prints `[✓] migrations applied successfully!`. Exit `0`. The third migration's hash is never inserted into `__drizzle_migrations`.
A standalone reproducer (Node + `pg` against a local Postgres) that toggles a single row to flip the outcome can be provided on request.
### Why this matters
The silent-skip is hard to detect in CI because:
- The exit code is `0`.
- No log line names the skipped migration.
- The next `drizzle-kit migrate` call (e.g. on the next deploy) still exits `0`.
- The schema drift only surfaces when a query at runtime hits a column or table the skipped migration was supposed to add — usually in production, not in CI.
Several real-world ops runbooks for "drizzle migration journal drift recovery" recommend the manual `INSERT … (hash, created_at) VALUES (…, now())` pattern. Those recipes are silently broken under the current watermark logic.
### Suggested fixes
Either of these resolves the correctness problem:
1. **Apply-by-hash-absence.** Replace the `MAX(created_at)` watermark with `EXISTS (SELECT 1 FROM __drizzle_migrations WHERE hash = $1)` per pending journal entry. To preserve the cheap single-query startup, fetch all `hash` values once into a `Set` and check presence in a hash set. Cost: one extra column projected and a slightly larger result set; the migration table is small in practice.
2. **Document the watermark invariant + add a runtime check.** Keep the watermark for performance, but on every skipped journal entry, look up `EXISTS (SELECT 1 FROM __drizzle_migrations WHERE hash = $expected_hash)` and `throw` (or at minimum `console.warn`) if the hash isn't present. This preserves the cheap path and surfaces the bug loudly when it triggers.
Option 1 is the cleaner fix because it removes the "manual repair invalidates the watermark" foot-gun entirely. Option 2 is the smaller diff if backwards-compatibility for downstream consumers depending on the current behaviour is a concern.
### Environment
- `drizzle-orm`: tested up to current published version on npm (also reviewed `main` branch source — predicate unchanged).
- `drizzle-kit`: any published version that uses `PgDialect.migrate`.
- Drivers: reproduces on `neon-serverless`, `neon-http`, `node-postgres`, `postgres-js` (all delegate to the same `PgDialect.migrate` predicate).
### Workaround we're using
Until this is fixed upstream, we run a small runner (~80 lines of `tsx` + `pg`) that reads `drizzle/meta/_journal.json` and applies any entry whose `hash` is NOT present in `__drizzle_migrations`, regardless of `created_at`. We'd happily retire it once the upstream predicate is fixed — hence this issue.
Happy to send a minimal reproducer repo or a draft PR if either would speed things along.
Contributor guide
Assessment
This issue has not been assessed yet.