[core] Parked-chain delivery order across drain windows relies on unpinned safety-net FIFO
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Summary
Log-order delivery for a chain parked behind an unclaimed hook payload relies on the payload's idle safety net observing idle before the net of the wait parked behind it. Within one drain window that order is structural. Across drain windows it is not pinned, and nothing currently guarantees it.
Raised by @pranaygp reviewing #3406 (https://github.com/vercel/workflow/pull/3406#discussion_r3752349996); filing so it does not live only in a review thread. #3406 documents the guarantee and its scope, and does not change this behavior.
Why the order matters
A hook payload registers its delivery barrier unarmed when no branch is waiting on it (workflow/hook.ts, armed: promises.length > 0), so only the barrier registry's idle safety net can retire it. A wait ordered after it in the log parks behind it, and a step result gates on that wait (#3406). Correct unwinding is payload -> wait -> step.
If the wait's net fired first, the wait would retire before the payload delivered, the step's gate would open while the wait was still parked, and the two branches would swap the correlation ids they draw next: the ReplayDivergenceError / CORRUPTED_EVENT_LOG shape #3406 fixes.
Where it holds, and where it does not
Within one drain window the order is carried by FIFO of the safety-net polls: nets arm via setTimeout in log order during synchronous consumption, each polling round re-arms through promiseQueue.then(...) in the order the checks ran, and each net that fires flips hasParkedCommittedDelivery back to true, re-blocking the rest until the released delivery completes. Replay always consumes the log in one window, which is where divergence manifests, so the fixed shape is covered.
The gap is a live invocation whose unclaimed-payload barrier survives from an earlier drain window (the run never went idle in between). That loop's timer-queue position relative to nets armed in a later window depends on when its previously-subscribed queue tail resolves, so payload-before-wait is not pinned. This predates #3406 — main has the same structure for hook-vs-wait ordering — and #3406's fix does not depend on it.
Why it may not be academic
stepStormReproWorkflow is exactly this shape: a poke hook that is never read, so every hook_received arrives unclaimed, racing steps against watchdog sleeps over a long-lived run. #3406 takes that scenario from 8/18 to 0/18 CORRUPTED_EVENT_LOG locally against @workflow/world-postgres, so the dominant cause is addressed. Worth checking whether any residual corruption at higher soak scale lands on this ordering rather than something else.
Suggested next steps
- Determine whether a cross-window parked payload is reachable in practice, or whether an unclaimed payload's net always fires in the window that registered it (in which case the concern is structural-only and the doc note in #3406 is sufficient).
- If reachable, make the ordering explicit rather than emergent, e.g. retire parked entries in log-index order from a single net rather than relying on independent per-barrier timers racing.
- Soak
step-stormat the historical scale (EVENT_LOG_RACE_REPRO_STEP_STORM_ATTEMPTS=600) on top of #3406 and attribute anything that still corrupts.
Note for whoever picks this up: hasParkedCommittedDelivery's early return means the recursive registry walk cannot be exponential, so the memo on resolvesOnItsOwn is an optimization only (measured in #3406). Do not treat the walk's cost as a constraint on a fix here.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with workflow/hook.ts and the guarantee documented in #3406. Investigate whether a parked payload can survive into a later drain window, then run step-storm at EVENT_LOG_RACE_REPRO_STEP_STORM_ATTEMPTS=600 and attribute any remaining corruption. Done means either documenting the structural-only concern or making cross-window delivery order explicit and verifying the soak behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres, typescript
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100