vercel / vercel/workflow

CORRUPTED_EVENT_LOG on @workflow/core 4.6.1 with Promise.race([Promise.all(multiple hooks), sleep()])

Open
#3,070 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Summary

A production Vercel Workflow using @workflow/core@4.6.1 repeatedly fails during recovery replay with CORRUPTED_EVENT_LOG when an ordered batch of multiple hooks is raced against sleep():

const hooks = children.map(() => createHook<Result>());

const outcome = await Promise.race([
  Promise.all(hooks.map(async (hook) => await hook)),
  sleep("35m"),
]);

The run made normal forward progress for approximately 41 minutes before a recovery replay diverged. The runtime attempted three recovery replays and then marked the run failed after the fourth divergence.

This looks related to #2171, but appears to reproduce on @workflow/core@4.6.1 with the multi-hook Promise.all(...) variant.

Actual behavior

The terminal workflow error was:

CORRUPTED_EVENT_LOG
Workflow replay diverged 4 times after 3 recovery replays.
Last divergence: Replay could not consume event:
eventType=hook_created,
correlationId=<hook-id>,
eventId=<event-id>.

The workflow's application/domain ledger remained in its last running state because the provider failed before finalization.

Expected behavior

Recovery replay should consume the committed hook and wait events in deterministic event-log order and continue from the last durable checkpoint.

Environment

  • Vercel Workflows, production
  • Region: iad1
  • workflow: 4.6.1
  • @workflow/core: 4.6.1
  • @workflow/world-vercel: 4.5.1
  • @workflow/next: 4.1.1
  • Next.js: 16.2.4
  • Workflow spec version shown in the trace: 3
  • Batch concurrency: 8

Additional context

The original application used a sliding worker pool. We first removed that source of nondeterminism and changed it to fixed, ledger-ordered batches: hooks are now created sequentially in child order, each queue step is awaited, and the batch does not advance until the current batch settles.

The remaining orchestration primitive is effectively:

for (const batch of orderedBatches(children, 8)) {
  const hooks = [];

  for (const child of batch) {
    const hook = createHook<Result>();
    hooks.push({ child, hook });
    await queueChildStep(child, hook.token);
  }

  await Promise.race([
    Promise.all(hooks.map(({ hook }) => hook)),
    sleep("35m"),
  ]);
}

The failure happened after many batches had completed successfully, during a provider recovery replay. The divergent event was hook_created, not an application step failure, queue delivery error, or hook payload validation error.

PR #2171 describes a hook-versus-sleep replay race caused by delivery/microtask ordering. Since this run used @workflow/core@4.6.1, I suspect either:

  1. the multiple-hook Promise.all shape exposes an uncovered variant, or
  2. there is a regression in the later runtime.

I can provide the private Vercel workflow run ID, deployment ID, exact event ID, correlation ID, and full trace directly to a maintainer through a non-public channel.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the createHook, sleep, Promise.race, and Promise.all entry points described in the reproduction, then trace how recovery replay consumes hook_created events. Reproduce the fixed-batch scenario if possible and compare normal execution with recovery replay; done means committed hook and wait events are consumed deterministically without CORRUPTED_EVENT_LOG.

Written by the indexing model from the issue text.

Assessment

Tech stack
next.js, typescript
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.