vercel / vercel/workflow

world-postgres: complete the wait row and its wait_completed event atomically

Open
#4,210 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Package: @workflow/world-postgres@5.0.0-beta.44 (@workflow/world@5.0.0-beta.36), Node 24, PostgreSQL 17. First observed on beta.40; the code is unchanged on main today.

wait_completed mutates the wait row and writes its event in two separate autocommits, so a failure between them leaves a wait that can never be completed — and the runtime turns that into an unbounded continuation loop rather than an error.

In packages/world-postgres/src/storage.ts the wait_completed branch does:

UPDATE workflow_waits SET status='completed'   -- commits
INSERT workflow_events (wait_completed)        -- commits later, in the shared tail

If the second statement does not land, the row is completed while the log has no wait_completed event. Every later completion attempt now takes the else branch, finds existing.status === 'completed' and throws EntityConflictError("Wait ... already completed"). The runtime swallows that conflict, still observes an elapsed, uncompleted wait on replay, and schedules another continuation with a fresh idempotency key — about once a second, forever.

We hit this in production on 2026-09-11. Two runs in this state produced ~1700 queue requests per minute and roughly 27 parallel job chains each, with the wait-continuation attempt counter passing 26 000. Both step-runner pods OOMed every few minutes and worker concurrency was fully occupied, so unrelated work backed up behind it. The runs themselves looked healthy: workflow_runs was running, the wait row was completed, and only the wait_completed event was missing. Recovery required cancelling the runs by hand; nothing in the library can heal the state on its own.

This is the same "mutate entity, then write event" shape as #3081 (step_created) and #3656 (run creation), both of which are now transactional on main. wait_completed is the remaining instance.

Suggested fix, mirroring what step_created already does:

  1. Run the UPDATE and the event INSERT inside one drizzle.transaction(..., SLOT_INSERT_TRANSACTION), and skip the shared tail insertion for this event type.
  2. When the guarded UPDATE matches nothing, re-read the row FOR UPDATE inside that transaction. If it is already completed, look for the matching wait_completed event: only raise EntityConflictError when the event exists, otherwise treat the row as an orphan from the old path and write the missing event. That keeps duplicate completions rejected while letting existing orphans heal on the next attempt instead of looping.

The invariant should be: a completed wait row and its wait_completed event become visible together, and a wait row can never be completed without one.

We carry exactly this as a local Bun patch, with a real-Postgres regression test (orphan row healed; duplicate completion still rejected). Happy to open a PR if that shape is acceptable.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/world-postgres/src/storage.ts and compare the wait_completed branch with the transactional step_created path on main. Add a real-Postgres regression test covering orphan-row healing and duplicate completion rejection. Done means the completed wait row and wait_completed event become visible together, with existing orphans recoverable on the next attempt.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, typescript
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.