CORRUPTED_EVENT_LOG on @workflow/core 4.6.1 with Promise.race([Promise.all(multiple hooks), sleep()])
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Summary
A production Vercel Workflow using @workflow/core@4.6.1 repeatedly fails during recovery replay with CORRUPTED_EVENT_LOG when an ordered batch of multiple hooks is raced against sleep():
const hooks = children.map(() => createHook<Result>());
const outcome = await Promise.race([
Promise.all(hooks.map(async (hook) => await hook)),
sleep("35m"),
]);
The run made normal forward progress for approximately 41 minutes before a recovery replay diverged. The runtime attempted three recovery replays and then marked the run failed after the fourth divergence.
This looks related to #2171, but appears to reproduce on @workflow/core@4.6.1 with the multi-hook Promise.all(...) variant.
Actual behavior
The terminal workflow error was:
CORRUPTED_EVENT_LOG
Workflow replay diverged 4 times after 3 recovery replays.
Last divergence: Replay could not consume event:
eventType=hook_created,
correlationId=<hook-id>,
eventId=<event-id>.
The workflow's application/domain ledger remained in its last running state because the provider failed before finalization.
Expected behavior
Recovery replay should consume the committed hook and wait events in deterministic event-log order and continue from the last durable checkpoint.
Environment
- Vercel Workflows, production
- Region:
iad1 workflow:4.6.1@workflow/core:4.6.1@workflow/world-vercel:4.5.1@workflow/next:4.1.1- Next.js:
16.2.4 - Workflow spec version shown in the trace:
3 - Batch concurrency:
8
Additional context
The original application used a sliding worker pool. We first removed that source of nondeterminism and changed it to fixed, ledger-ordered batches: hooks are now created sequentially in child order, each queue step is awaited, and the batch does not advance until the current batch settles.
The remaining orchestration primitive is effectively:
for (const batch of orderedBatches(children, 8)) {
const hooks = [];
for (const child of batch) {
const hook = createHook<Result>();
hooks.push({ child, hook });
await queueChildStep(child, hook.token);
}
await Promise.race([
Promise.all(hooks.map(({ hook }) => hook)),
sleep("35m"),
]);
}
The failure happened after many batches had completed successfully, during a provider recovery replay. The divergent event was hook_created, not an application step failure, queue delivery error, or hook payload validation error.
PR #2171 describes a hook-versus-sleep replay race caused by delivery/microtask ordering. Since this run used @workflow/core@4.6.1, I suspect either:
- the multiple-hook
Promise.allshape exposes an uncovered variant, or - there is a regression in the later runtime.
I can provide the private Vercel workflow run ID, deployment ID, exact event ID, correlation ID, and full trace directly to a maintainer through a non-public channel.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the createHook, sleep, Promise.race, and Promise.all entry points described in the reproduction, then trace how recovery replay consumes hook_created events. Reproduce the fixed-batch scenario if possible and compare normal execution with recovery replay; done means committed hook and wait events are consumed deterministically without CORRUPTED_EVENT_LOG.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- next.js, typescript
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100