vercel / vercel/workflow

webhookWorkflow times out on '- stable' local e2e lanes: run stalls in 'running' after all hooks received

Open
#3,317 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Stable-lane e2e jobs (E2E Local Dev/Prod Tests (<app> - stable)) are failing frequently across apps with e2e > webhookWorkflow timing out at 120s. Every main push run today (2026-08-03/04) failed at least one such lane on it — apps rotate (astro, sveltekit, nextjs-webpack, express, nest, nitro), astro most often. Examples: 30866461896, 30863551393, 30829099358. On PR #3314 (no package-code changes, based on main @ 89ede82fa) astro - stable failed on it 5 rerun attempts in a row — deterministic there, while other apps' lanes pass/fail intermittently.

Note on naming: - stable in these job names is the framework toolchain label from scripts/create-test-matrix.mjs (runLabel: canary ? 'canary' : 'stable'), not the stable release channel — these jobs run the repo's current packages (job logs show Workflow CLI v5.0.0-beta.39). So this is a hang in current main code.

Failure shape

The run gets all the way through hook delivery and the built-in response steps, then stalls at running forever (from the vitest diagnostics on attempt 4, run wrun_01KZ58CGGNW6HP56G9B1K7T0SR):

+0.0s  run_created
+0.0s  run_started
+0.1s  hook_created  x3
+1.0s  hook_received [..W]
+1.1s  hook_received [..X]
+1.1s  step_started/created (__builtin_response_text) [..1Z]
+1.2s  hook_received [..Y]
+1.2s  step_completed (__builtin_response_text) [..1Z]
+1.3s  step_started/created (__builtin_response_text) [..20]
+1.4s  step_completed (__builtin_response_text) [..20]
(nothing further; status stays 'running' until the 120s test timeout)

All three webhooks (default / static / manual respondWith) receive their payloads and both response-body steps complete, but the workflow never progresses to completion. webhookWorkflow and its workflow source were last touched well before today, so the trigger is likely a recent runtime change (the VM-retention work #3046/#3047 landed in this window, though failures appear in at least one run predating them — worth a bisect with WORKFLOW_RETAINED_VM=0 as a first probe).

🤖 Generated with Claude Code

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The failure is exercised by e2e > webhookWorkflow, with matrix labels in scripts/create-test-matrix.mjs; start by reproducing the astro - stable lane and inspecting the listed vitest diagnostics. Use WORKFLOW_RETAINED_VM=0 as the first comparison or bisect probe; done means webhookWorkflow reaches completion and the affected stable lanes no longer time out.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.