world-postgres: complete the wait row and its wait_completed event atomically
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Package: @workflow/world-postgres@5.0.0-beta.44 (@workflow/world@5.0.0-beta.36), Node 24, PostgreSQL 17. First observed on beta.40; the code is unchanged on main today.
wait_completed mutates the wait row and writes its event in two separate autocommits, so a failure between them leaves a wait that can never be completed — and the runtime turns that into an unbounded continuation loop rather than an error.
In packages/world-postgres/src/storage.ts the wait_completed branch does:
UPDATE workflow_waits SET status='completed' -- commits
INSERT workflow_events (wait_completed) -- commits later, in the shared tail
If the second statement does not land, the row is completed while the log has no wait_completed event. Every later completion attempt now takes the else branch, finds existing.status === 'completed' and throws EntityConflictError("Wait ... already completed"). The runtime swallows that conflict, still observes an elapsed, uncompleted wait on replay, and schedules another continuation with a fresh idempotency key — about once a second, forever.
We hit this in production on 2026-09-11. Two runs in this state produced ~1700 queue requests per minute and roughly 27 parallel job chains each, with the wait-continuation attempt counter passing 26 000. Both step-runner pods OOMed every few minutes and worker concurrency was fully occupied, so unrelated work backed up behind it. The runs themselves looked healthy: workflow_runs was running, the wait row was completed, and only the wait_completed event was missing. Recovery required cancelling the runs by hand; nothing in the library can heal the state on its own.
This is the same "mutate entity, then write event" shape as #3081 (step_created) and #3656 (run creation), both of which are now transactional on main. wait_completed is the remaining instance.
Suggested fix, mirroring what step_created already does:
- Run the
UPDATEand the eventINSERTinside onedrizzle.transaction(..., SLOT_INSERT_TRANSACTION), and skip the shared tail insertion for this event type. - When the guarded
UPDATEmatches nothing, re-read the rowFOR UPDATEinside that transaction. If it is alreadycompleted, look for the matchingwait_completedevent: only raiseEntityConflictErrorwhen the event exists, otherwise treat the row as an orphan from the old path and write the missing event. That keeps duplicate completions rejected while letting existing orphans heal on the next attempt instead of looping.
The invariant should be: a completed wait row and its wait_completed event become visible together, and a wait row can never be completed without one.
We carry exactly this as a local Bun patch, with a real-Postgres regression test (orphan row healed; duplicate completion still rejected). Happy to open a PR if that shape is acceptable.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in packages/world-postgres/src/storage.ts and compare the wait_completed branch with the transactional step_created path on main. Add a real-Postgres regression test covering orphan-row healing and duplicate completion rejection. Done means the completed wait row and wait_completed event become visible together, with existing orphans recoverable on the next attempt.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, typescript
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100