Vercel World queue message waited 39m for first delivery with no redelivery
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Turn Workflow first-delivery stall
Date investigated: 2026-08-23
Conclusion
An iMessage turn waited 2,343.577 seconds between Eve's successful child-run
dispatch and turn.started. The delay occurred before the child Workflow
consumer ran. Vercel Queues recorded one first delivery after the message had
aged for 39 minutes, with no redelivery in the incident window.
This was not model execution, compaction, session size, per-thread turn
serialization, a failed consumer attempt, or deployment churn. It is a
Vercel Queues first-delivery tail affecting Workflow's Vercel World.
Incident timeline
All timestamps are UTC on 2026-08-22.
| Event | Time |
|---|---|
| Photon webhook accepted | 22:46:04 |
| Eve parent received the turn | 22:46:08 |
dispatchTurnStep completed |
22:46:14.852 |
Child turn.started |
23:25:18.429 |
The child model and tool execution took another 67.78 seconds after it began;
that work is outside the 39-minute gap.
For the exact incident window, the turnWorkflow consumer showed 31 messages
received, 31 deleted, zero redeliveries, retry depth 1, p50 delivery age 0 ms,
and maximum delivery age 39 minutes. A visibility-timeout retry would increase
both delivery count and redeliveries; neither happened.
Cohort evidence
Across 88 historical dispatch-to-start samples for the same production user:
- median: 4.551 seconds
- p90: 18.082 seconds
- p95: 32.673 seconds
- next worst: 57.767 seconds
- incident: 2,343.577 seconds
Vercel's seven-day queue view showed:
| Queue | Received | Deleted | Redeliveries | p50 | p95 | p99 | Max |
|---|---|---|---|---|---|---|---|
turnWorkflow/default |
825 | 807 | 18 | 0 ms | 1 s | 35 min | 2 h |
workflowEntry/default |
770 | 770 | 0 | 0 ms | 1 s | 1 s | 14 min |
All 18 turnWorkflow redeliveries in the seven-day range occurred in the
latest 12-hour slice, where maximum retry depth reached 19. The incident itself
was not one of those retries. The zero-redelivery sibling queue's 14-minute
maximum independently shows a first-delivery tail.
Queue traffic averaged roughly 0.0012 messages per second. Other turns ran
while the incident child was pending, so neither throughput saturation nor a
globally unavailable consumer explains one message being skipped for 39
minutes.
Runtime and configuration
- eve 0.44.0
@workflow/core5.0.0-beta.43 (vendored by eve)@workflow/world-vercel5.0.0-beta.39 (vendored by eve)- generated queue trigger:
initialDelaySeconds: 0,
retryAfterSeconds: 5, consumerdefault - no app-configured consumer concurrency limit
- the child start targets deployment
latest - no deployment occurred during the delayed interval
- incident-window runtime requests returned HTTP 200
App-side guardrail
The app now stamps the Photon acceptance time into turn auth and records
delivery.inboundToTurnMs in the evlog wide event at turn.started. Values at
or above 60 seconds are force-kept and warned as slow_turn_start.
The metric is deliberately not called queue latency. With turnPolicy: "queue"
it may include legitimate waiting behind an active turn, parent Workflow
dispatch, child queue delivery, and child bootstrap. Vercel's message
createdAt/delivery-age telemetry remains authoritative for platform
attribution.
Upstream questions
- What can cause a visible, zero-delay message to receive no first consumer
invocation for 39 minutes while newer messages on the same queue are
delivered? - Is there a first-delivery latency objective, and can Workflow surface a
per-runcreatedAttoconsumerStartedAtmeasurement so this is diagnosable
without correlating three dashboards? - Could per-deployment partitioning or consumer-registration state starve an
individual message even when the consumer has no configured concurrency
cap?
Exact project, queue, deployment, and run identifiers can be shared privately
with Vercel staff.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files or tests are named. Start with the generated queue trigger and the vendored @workflow/core 5.0.0-beta.43 and @workflow/world-vercel 5.0.0-beta.39 components, then correlate queue createdAt, delivery, and consumer-start events for the incident window. Done means explaining the first-delivery stall or identifying a concrete upstream change needed to diagnose or prevent it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100