Detect and recover stranded hook resumes (committed hook_received, no delivered wake)
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Follow-up from #3841 (durable hook resume) review discussion.
With the serial dispatch, resumeHook() only resolves after the durable hook_received write AND an accepted wake publish. Two stranded shapes remain possible, both leaving a committed event that nothing re-drives:
- Wake publish failed after the commit — the caller gets an error (observable), but the event sits in the log until some other wake happens to arrive.
- Wake published (200) but never delivered — queue loss, retry exhaustion, or consumer discovery failure against a gone deployment. No signal anywhere: the producer saw success, and the
*/5stuck-runs cron is metrics-only (gauges from ClickHouse, no re-drive).
#3841 ships detection primitives: workflow.hook.resume_committed and workflow.hook.wake_published span attributes stamped after each leg resolves, so shape 1 is directly queryable (resume_committed=true without wake_published), and shape 2 is queryable by correlation (wake_published=true for run R with no subsequent workflow execution span for R within N seconds).
Remaining work:
- Consumer-side correlation: emit the consumed
resumeId(s) on the replay span whenhook_receivedevents are delivered into the VM, so the join is exact rather than runId+time-window based. - Alerting: back a ClickHouse alert with the committed-without-consumption query, alongside where the stuck-runs cron already looks.
- Recovery (choose one):
- a delayed safety wake published alongside the real one (replay is idempotent; the per-run topic serializes them;
Vqs-Delay-Secondsexists), or - teach the stuck-runs cron to re-publish a wake for runs whose newest event is a
hook_receivedwith no subsequent progress.
- a delayed safety wake published alongside the real one (replay is idempotent; the per-run topic serializes them;
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with resumeHook() and the existing workflow.hook.resume_committed and workflow.hook.wake_published span attributes described in the issue. Trace the replay span, consumed resumeId values, the ClickHouse alert, and the stuck-runs cron. Done means consumption correlation is exact, stranded commits are alerted, and one idempotent recovery path re-drives them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100