Workflow engine silently stops executing runs after uptime: webhook/manual runs stay pending forever, schedules never fire (unsupervised engine task)
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
On a hosted community relay, the workflow engine **silently stops executing runs entirely** after some uptime. Every trigger path is affected once it happens: webhook-triggered runs and manual triggers return `202` / `accepted` with a `run_id` and status `pending`, but the run never executes and no step actions happen. Scheduled (cron/interval) triggers are the most visible casualty because they depend on the engine loop end to end, which likely explains #5611 and #4904. #4288 (message_posted dead while others work) looks like the same class of failure caught at a different moment.
The HTTP/ingest side keeps accepting work the whole time, so from the operator's point of view everything reports success while nothing ever runs.
## Environment
- Hosted community relay: `jj.communities.buzz.xyz` (communities.buzz.xyz cloud)
- Observed 2026-08-19 through 2026-08-21
- Source read at `4e3c9e619c93dd26677b392ad1f8cf0d12c8f855`
## Timeline of live tests (all on 2026-08-21, times UTC)
A dedicated throwaway channel was used, with four minimal one-step (`send_message`) workflows, one per trigger type.
**Window A (engine alive, ~15:48–15:55):**
- `message_posted` trigger: fired, message posted within ~1s
- manual trigger: fired, message posted
- webhook trigger (`POST /hooks/{id}` with `x-webhook-secret`): fired, message posted in <5s
- cron `* * * * *` and interval `60s`: fired **once** (15:48:41), then never again over 8+ minutes of watching
**Window B (same relay, same channel, ~30 minutes later, 16:20+):**
- webhook trigger on a freshly created enabled workflow: `HTTP 202`, `{"run_id":"…","status":"pending"}` — never executed: no message landed even 15+ minutes later (vs <5s in Window A)
- manual trigger (`workflows trigger`) on the same workflow: `accepted` with a `run_id` — never executed, no message ever landed
Nothing about the workflow changed between windows; the engine simply stopped processing. Two days earlier (2026-08-19) another operator on the same relay saw the same thing: manual trigger `accepted` + `run_id`, no execution, no resulting action.
## Suspected cause
The engine loop is a single unsupervised task spawned once at startup:
```rust
// crates/buzz-relay/src/main.rs:658
let wf_cron = Arc::clone(&workflow_engine);
tokio::spawn(async move { wf_cron.run().await });
```
If `run()` returns an error or the task panics, there is no supervisor, no restart, no health signal, and no alarm. Handlers keep creating `pending` runs that nothing will ever pick up, which matches the observed "202 + run_id + eternal pending" exactly. The one-fire-then-silence cron behavior in Window A also fits a loop that died mid-tick.
## Suggested fixes
1. Supervise the engine task: restart-on-exit with backoff, and log loudly on any exit.
2. Expose a liveness/health signal for the engine loop (metric or admin endpoint) so a dead engine is observable instead of silent.
3. On engine (re)start, sweep and requeue orphaned `pending` runs so accepted work is not lost forever.
4. Short term: restarting the hosted relay process restores execution (Window A behavior is consistent with a recent restart).
## Repro
1. Create a minimal enabled workflow with `trigger: {on: webhook}` and one `send_message` step.
2. `POST /hooks/{workflow_id}` with the secret → observe `202` + `pending`.
3. If the relay's engine loop has died (give a long-uptime relay time, or kill the task), the run never executes and the endpoint keeps returning `202` while nothing ever runs.
Contributor guide
Research direction
Start at crates/buzz-relay/src/main.rs:658 and inspect how the spawned workflow engine task runs, exits, and reports failures. Reproduce the dead-task behavior with webhook, manual, and scheduled triggers, then trace how pending runs are created and consumed. Done should include a defined recovery path for engine failure, visibility into liveness, and no permanently pending accepted runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100