Local world: 30s undici transport timeout redelivers a live queue message and re-executes an in-flight lazy-inline step
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Package: @workflow/world-local (eve 0.38.3 vendors this; eve 0.47.6 still vendors 5.0.0-beta.37 with the same default)
Area: local World queue delivery over undici; @workflow/core recovery of lazy-inline steps
What happens
The local world delivers a queue message by POSTing to the workflow route and waiting for the handler to finish. For a "use step" lazy-inline body, that POST stays open for the whole step (a model call can run for minutes).
The undici Agent is built as:
headersTimeout: yn("WORKFLOW_LOCAL_HEADERS_TIMEOUT_MS", 30000)
bodyTimeout: yn("WORKFLOW_LOCAL_BODY_TIMEOUT_MS", 30000)
(yn treats unset/empty as the fallback; 0 is accepted and disables the timer.) After 30s the client throws TypeError: fetch failed. The queue logs a transport failure, sleeps ~5s, and redelivers the same message. Recovery then re-executes the still-running inline step.
eve 0.45.2 (5934d69) claimed to “Prevent long local Workflow deliveries from timing out and replaying an in-flight turn.” The 0.47.6 vendored world still defaults both timeouts to 30000. Explicit env overrides still take precedence.
Evidence pattern
- Log:
[world-local] Queue delivery failed at the transport (loop N), retryingwithTypeError: fetch failed(~30s after the delivery POST). - Log:
[workflow-sdk] Re-executing inline steps owned by this queue message(wording may include “a previous delivery crashed mid-body”). - Step journal under the local data dir (
WORKFLOW_LOCAL_DATA_DIR/.workflow-data/steps/*.json): the same step id records"attempt": 2while the first attempt is still live. - Observable effect: the step’s side effects run twice (duplicate model calls / duplicate writes).
Single-flight appears to cover only the recovery path, so both branches run.
Expected
A local delivery whose handler is still alive must not be treated as a crash. Options: disable headers/body timeouts by default for local queue POSTs; treat a still-open in-process handler as ownership of the message; or fail the redelivery while the original socket is open.
Workaround
WORKFLOW_LOCAL_HEADERS_TIMEOUT_MS=0
WORKFLOW_LOCAL_BODY_TIMEOUT_MS=0
undici documents 0 as “disable entirely.” A hung worker that actually dies still closes the socket and retries. A large finite ceiling only moves the cliff.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in @workflow/world-local queue delivery and @workflow/core lazy-inline recovery, tracing the undici timeout defaults, transport retry, and ownership checks. Reproduce with a long-running lazy-inline step and inspect the step journal under WORKFLOW_LOCAL_DATA_DIR; done means a live handler is not redelivered or re-executed, while a dead worker still retries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100