vercel / vercel/workflow

Detect and recover stranded hook resumes (committed hook_received, no delivered wake)

Open
#3,911 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Follow-up from #3841 (durable hook resume) review discussion.

With the serial dispatch, resumeHook() only resolves after the durable hook_received write AND an accepted wake publish. Two stranded shapes remain possible, both leaving a committed event that nothing re-drives:

  1. Wake publish failed after the commit — the caller gets an error (observable), but the event sits in the log until some other wake happens to arrive.
  2. Wake published (200) but never delivered — queue loss, retry exhaustion, or consumer discovery failure against a gone deployment. No signal anywhere: the producer saw success, and the */5 stuck-runs cron is metrics-only (gauges from ClickHouse, no re-drive).

#3841 ships detection primitives: workflow.hook.resume_committed and workflow.hook.wake_published span attributes stamped after each leg resolves, so shape 1 is directly queryable (resume_committed=true without wake_published), and shape 2 is queryable by correlation (wake_published=true for run R with no subsequent workflow execution span for R within N seconds).

Remaining work:

  • Consumer-side correlation: emit the consumed resumeId(s) on the replay span when hook_received events are delivered into the VM, so the join is exact rather than runId+time-window based.
  • Alerting: back a ClickHouse alert with the committed-without-consumption query, alongside where the stuck-runs cron already looks.
  • Recovery (choose one):
    • a delayed safety wake published alongside the real one (replay is idempotent; the per-run topic serializes them; Vqs-Delay-Seconds exists), or
    • teach the stuck-runs cron to re-publish a wake for runs whose newest event is a hook_received with no subsequent progress.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with resumeHook() and the existing workflow.hook.resume_committed and workflow.hook.wake_published span attributes described in the issue. Trace the replay span, consumed resumeId values, the ClickHouse alert, and the stuck-runs cron. Done means consumption correlation is exact, stranded commits are alerted, and one idempotent recovery path re-drives them.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.