vercel / vercel/workflow

Local world: 30s undici transport timeout redelivers a live queue message and re-executes an in-flight lazy-inline step

Open
#3,909 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Package: @workflow/world-local (eve 0.38.3 vendors this; eve 0.47.6 still vendors 5.0.0-beta.37 with the same default)
Area: local World queue delivery over undici; @workflow/core recovery of lazy-inline steps

What happens

The local world delivers a queue message by POSTing to the workflow route and waiting for the handler to finish. For a "use step" lazy-inline body, that POST stays open for the whole step (a model call can run for minutes).

The undici Agent is built as:

headersTimeout: yn("WORKFLOW_LOCAL_HEADERS_TIMEOUT_MS", 30000)
bodyTimeout:    yn("WORKFLOW_LOCAL_BODY_TIMEOUT_MS", 30000)

(yn treats unset/empty as the fallback; 0 is accepted and disables the timer.) After 30s the client throws TypeError: fetch failed. The queue logs a transport failure, sleeps ~5s, and redelivers the same message. Recovery then re-executes the still-running inline step.

eve 0.45.2 (5934d69) claimed to “Prevent long local Workflow deliveries from timing out and replaying an in-flight turn.” The 0.47.6 vendored world still defaults both timeouts to 30000. Explicit env overrides still take precedence.

Evidence pattern

  1. Log: [world-local] Queue delivery failed at the transport (loop N), retrying with TypeError: fetch failed (~30s after the delivery POST).
  2. Log: [workflow-sdk] Re-executing inline steps owned by this queue message (wording may include “a previous delivery crashed mid-body”).
  3. Step journal under the local data dir (WORKFLOW_LOCAL_DATA_DIR / .workflow-data/steps/*.json): the same step id records "attempt": 2 while the first attempt is still live.
  4. Observable effect: the step’s side effects run twice (duplicate model calls / duplicate writes).

Single-flight appears to cover only the recovery path, so both branches run.

Expected

A local delivery whose handler is still alive must not be treated as a crash. Options: disable headers/body timeouts by default for local queue POSTs; treat a still-open in-process handler as ownership of the message; or fail the redelivery while the original socket is open.

Workaround

WORKFLOW_LOCAL_HEADERS_TIMEOUT_MS=0
WORKFLOW_LOCAL_BODY_TIMEOUT_MS=0

undici documents 0 as “disable entirely.” A hung worker that actually dies still closes the socket and retries. A large finite ceiling only moves the cliff.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in @workflow/world-local queue delivery and @workflow/core lazy-inline recovery, tracing the undici timeout defaults, transport retry, and ownership checks. Reproduce with a long-running lazy-inline step and inspect the step journal under WORKFLOW_LOCAL_DATA_DIR; done means a live handler is not redelivered or re-executed, while a dead worker still retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.