drizzle-team / drizzle-team/drizzle-orm

drizzle-kit migrate silently skips pending migrations when MAX(created_at) in __drizzle_migrations exceeds the journal `when` of pending entries

Open
#5,769 2 comments 6 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
35.8k
Forks
1.6k
Avg merge
2d 7h
Merged PRs (30d)
4

Description

### Summary

`drizzle-kit migrate` (via `PgDialect.migrate` in `drizzle-orm/pg-core/dialect.js`, used by every Postgres driver — `neon-serverless`, `neon-http`, `node-postgres`, `postgres-js`, `vercel-postgres`) decides whether a journal entry has already been applied by selecting the row with the highest `created_at` from `drizzle.__drizzle_migrations` and skipping any pending journal entry whose `folderMillis` (i.e. `_journal.json` `when`) is ≤ that value. The skip is silent — no log, no warning, no non-zero exit. The CLI still prints `[✓] migrations applied successfully!` and exits `0`.

This is a correctness bug in any case where a row in `__drizzle_migrations` has a `created_at` that doesn't come from drizzle-kit's own happy-path insert. That includes:

- The official "init from an existing schema" workflow.
- Any manual repair (e.g. recovering from `_journal.json` ↔ table drift via `INSERT INTO drizzle.__drizzle_migrations (hash, created_at) VALUES (…, extract(epoch from now())*1000);`, a pattern documented across multiple ops runbooks and Stack Overflow answers).
- Any project that bootstraps `__drizzle_migrations` from an existing production schema before adopting drizzle-kit.

In all of these, `created_at` is wall-clock-derived rather than journal-derived, and the next `drizzle-kit migrate` invocation will silent-skip every pending migration whose `_journal.json` `when` is earlier than that wall-clock value — including migrations the operator has just generated and very much intends to apply.

### Source

`drizzle-orm/src/pg-core/dialect.ts` (compiled to `drizzle-orm/pg-core/dialect.js:56-71` in published builds), `PgDialect.migrate`. The relevant predicate is roughly:

```js
const lastDbMigration = await tx.execute<{ id: number; hash: string; created_at: string }>(
sql`select id, hash, created_at from ${migrationsTable} order by created_at desc limit 1`,
);
// ...
if (!lastDbMigration[0] || Number(lastDbMigration[0].created_at) < migration.folderMillis) {
// apply migration
}
```

The single-row `MAX(created_at)` watermark is the root cause: a single poisoned `created_at` masks every later journal entry whose `when` ≤ that value, even if no row with that entry's `hash` exists in the table.

### Repro (any Postgres-backed app using `drizzle-kit migrate`)

1. Bootstrap a fresh Postgres database (any of the supported drivers).
2. Generate two drizzle migrations a few minutes apart so `_journal.json` has two entries with distinct `when` values, e.g. `1700000000000` and `1700000060000`.
3. Apply only the first migration via `drizzle-kit migrate` (or `psql` + `INSERT INTO drizzle.__drizzle_migrations (hash, created_at) VALUES ('', 1700000000000);`). Verify the table has one row with `created_at = 1700000000000`.
4. Now insert a poisoned-watermark row to simulate the manual-repair / seed-from-existing pattern: `INSERT INTO drizzle.__drizzle_migrations (hash, created_at) VALUES ('manual-repair-marker', 9999999999999);` (or use `extract(epoch from now())*1000` for any wall-clock value > the second journal entry's `when`).
5. Generate a third migration, e.g. `_journal.json` `when = 1700000120000`.
6. Run `drizzle-kit migrate`.

**Expected:** the third migration applies, OR drizzle-kit errors out citing the poisoned row.
**Actual:** the third migration is silently skipped. CLI prints `[✓] migrations applied successfully!`. Exit `0`. The third migration's hash is never inserted into `__drizzle_migrations`.

A standalone reproducer (Node + `pg` against a local Postgres) that toggles a single row to flip the outcome can be provided on request.

### Why this matters

The silent-skip is hard to detect in CI because:

- The exit code is `0`.
- No log line names the skipped migration.
- The next `drizzle-kit migrate` call (e.g. on the next deploy) still exits `0`.
- The schema drift only surfaces when a query at runtime hits a column or table the skipped migration was supposed to add — usually in production, not in CI.

Several real-world ops runbooks for "drizzle migration journal drift recovery" recommend the manual `INSERT … (hash, created_at) VALUES (…, now())` pattern. Those recipes are silently broken under the current watermark logic.

### Suggested fixes

Either of these resolves the correctness problem:

1. **Apply-by-hash-absence.** Replace the `MAX(created_at)` watermark with `EXISTS (SELECT 1 FROM __drizzle_migrations WHERE hash = $1)` per pending journal entry. To preserve the cheap single-query startup, fetch all `hash` values once into a `Set` and check presence in a hash set. Cost: one extra column projected and a slightly larger result set; the migration table is small in practice.

2. **Document the watermark invariant + add a runtime check.** Keep the watermark for performance, but on every skipped journal entry, look up `EXISTS (SELECT 1 FROM __drizzle_migrations WHERE hash = $expected_hash)` and `throw` (or at minimum `console.warn`) if the hash isn't present. This preserves the cheap path and surfaces the bug loudly when it triggers.

Option 1 is the cleaner fix because it removes the "manual repair invalidates the watermark" foot-gun entirely. Option 2 is the smaller diff if backwards-compatibility for downstream consumers depending on the current behaviour is a concern.

### Environment

- `drizzle-orm`: tested up to current published version on npm (also reviewed `main` branch source — predicate unchanged).
- `drizzle-kit`: any published version that uses `PgDialect.migrate`.
- Drivers: reproduces on `neon-serverless`, `neon-http`, `node-postgres`, `postgres-js` (all delegate to the same `PgDialect.migrate` predicate).

### Workaround we're using

Until this is fixed upstream, we run a small runner (~80 lines of `tsx` + `pg`) that reads `drizzle/meta/_journal.json` and applies any entry whose `hash` is NOT present in `__drizzle_migrations`, regardless of `created_at`. We'd happily retire it once the upstream predicate is fixed — hence this issue.

Happy to send a minimal reproducer repo or a draft PR if either would speed things along.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.