vercel / vercel/workflow

Vercel World queue message waited 39m for first delivery with no redelivery

Open
#3,753 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Turn Workflow first-delivery stall

Date investigated: 2026-08-23

Conclusion

An iMessage turn waited 2,343.577 seconds between Eve's successful child-run
dispatch and turn.started. The delay occurred before the child Workflow
consumer ran. Vercel Queues recorded one first delivery after the message had
aged for 39 minutes, with no redelivery in the incident window.

This was not model execution, compaction, session size, per-thread turn
serialization, a failed consumer attempt, or deployment churn. It is a
Vercel Queues first-delivery tail affecting Workflow's Vercel World.

Incident timeline

All timestamps are UTC on 2026-08-22.

Event Time
Photon webhook accepted 22:46:04
Eve parent received the turn 22:46:08
dispatchTurnStep completed 22:46:14.852
Child turn.started 23:25:18.429

The child model and tool execution took another 67.78 seconds after it began;
that work is outside the 39-minute gap.

For the exact incident window, the turnWorkflow consumer showed 31 messages
received, 31 deleted, zero redeliveries, retry depth 1, p50 delivery age 0 ms,
and maximum delivery age 39 minutes. A visibility-timeout retry would increase
both delivery count and redeliveries; neither happened.

Cohort evidence

Across 88 historical dispatch-to-start samples for the same production user:

  • median: 4.551 seconds
  • p90: 18.082 seconds
  • p95: 32.673 seconds
  • next worst: 57.767 seconds
  • incident: 2,343.577 seconds

Vercel's seven-day queue view showed:

Queue Received Deleted Redeliveries p50 p95 p99 Max
turnWorkflow/default 825 807 18 0 ms 1 s 35 min 2 h
workflowEntry/default 770 770 0 0 ms 1 s 1 s 14 min

All 18 turnWorkflow redeliveries in the seven-day range occurred in the
latest 12-hour slice, where maximum retry depth reached 19. The incident itself
was not one of those retries. The zero-redelivery sibling queue's 14-minute
maximum independently shows a first-delivery tail.

Queue traffic averaged roughly 0.0012 messages per second. Other turns ran
while the incident child was pending, so neither throughput saturation nor a
globally unavailable consumer explains one message being skipped for 39
minutes.

Runtime and configuration

  • eve 0.44.0
  • @workflow/core 5.0.0-beta.43 (vendored by eve)
  • @workflow/world-vercel 5.0.0-beta.39 (vendored by eve)
  • generated queue trigger: initialDelaySeconds: 0,
    retryAfterSeconds: 5, consumer default
  • no app-configured consumer concurrency limit
  • the child start targets deployment latest
  • no deployment occurred during the delayed interval
  • incident-window runtime requests returned HTTP 200

App-side guardrail

The app now stamps the Photon acceptance time into turn auth and records
delivery.inboundToTurnMs in the evlog wide event at turn.started. Values at
or above 60 seconds are force-kept and warned as slow_turn_start.

The metric is deliberately not called queue latency. With turnPolicy: "queue"
it may include legitimate waiting behind an active turn, parent Workflow
dispatch, child queue delivery, and child bootstrap. Vercel's message
createdAt/delivery-age telemetry remains authoritative for platform
attribution.

Upstream questions

  1. What can cause a visible, zero-delay message to receive no first consumer
    invocation for 39 minutes while newer messages on the same queue are
    delivered?
  2. Is there a first-delivery latency objective, and can Workflow surface a
    per-run createdAt to consumerStartedAt measurement so this is diagnosable
    without correlating three dashboards?
  3. Could per-deployment partitioning or consumer-registration state starve an
    individual message even when the consumer has no configured concurrency
    cap?

Exact project, queue, deployment, and run identifiers can be shared privately
with Vercel staff.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source files or tests are named. Start with the generated queue trigger and the vendored @workflow/core 5.0.0-beta.43 and @workflow/world-vercel 5.0.0-beta.39 components, then correlate queue createdAt, delivery, and consumer-start events for the incident window. Done means explaining the first-delivery stall or identifying a concrete upstream change needed to diagnose or prevent it.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.