block / block/buzz

Workflow engine silently stops executing runs after uptime: webhook/manual runs stay pending forever, schedules never fire (unsupervised engine task)

Open
#6,490 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

On a hosted community relay, the workflow engine **silently stops executing runs entirely** after some uptime. Every trigger path is affected once it happens: webhook-triggered runs and manual triggers return `202` / `accepted` with a `run_id` and status `pending`, but the run never executes and no step actions happen. Scheduled (cron/interval) triggers are the most visible casualty because they depend on the engine loop end to end, which likely explains #5611 and #4904. #4288 (message_posted dead while others work) looks like the same class of failure caught at a different moment.

The HTTP/ingest side keeps accepting work the whole time, so from the operator's point of view everything reports success while nothing ever runs.

## Environment

- Hosted community relay: `jj.communities.buzz.xyz` (communities.buzz.xyz cloud)
- Observed 2026-08-19 through 2026-08-21
- Source read at `4e3c9e619c93dd26677b392ad1f8cf0d12c8f855`

## Timeline of live tests (all on 2026-08-21, times UTC)

A dedicated throwaway channel was used, with four minimal one-step (`send_message`) workflows, one per trigger type.

**Window A (engine alive, ~15:48–15:55):**
- `message_posted` trigger: fired, message posted within ~1s
- manual trigger: fired, message posted
- webhook trigger (`POST /hooks/{id}` with `x-webhook-secret`): fired, message posted in <5s
- cron `* * * * *` and interval `60s`: fired **once** (15:48:41), then never again over 8+ minutes of watching

**Window B (same relay, same channel, ~30 minutes later, 16:20+):**
- webhook trigger on a freshly created enabled workflow: `HTTP 202`, `{"run_id":"…","status":"pending"}` — never executed: no message landed even 15+ minutes later (vs <5s in Window A)
- manual trigger (`workflows trigger`) on the same workflow: `accepted` with a `run_id` — never executed, no message ever landed

Nothing about the workflow changed between windows; the engine simply stopped processing. Two days earlier (2026-08-19) another operator on the same relay saw the same thing: manual trigger `accepted` + `run_id`, no execution, no resulting action.

## Suspected cause

The engine loop is a single unsupervised task spawned once at startup:

```rust
// crates/buzz-relay/src/main.rs:658
let wf_cron = Arc::clone(&workflow_engine);
tokio::spawn(async move { wf_cron.run().await });
```

If `run()` returns an error or the task panics, there is no supervisor, no restart, no health signal, and no alarm. Handlers keep creating `pending` runs that nothing will ever pick up, which matches the observed "202 + run_id + eternal pending" exactly. The one-fire-then-silence cron behavior in Window A also fits a loop that died mid-tick.

## Suggested fixes

1. Supervise the engine task: restart-on-exit with backoff, and log loudly on any exit.
2. Expose a liveness/health signal for the engine loop (metric or admin endpoint) so a dead engine is observable instead of silent.
3. On engine (re)start, sweep and requeue orphaned `pending` runs so accepted work is not lost forever.
4. Short term: restarting the hosted relay process restores execution (Window A behavior is consistent with a recent restart).

## Repro

1. Create a minimal enabled workflow with `trigger: {on: webhook}` and one `send_message` step.
2. `POST /hooks/{workflow_id}` with the secret → observe `202` + `pending`.
3. If the relay's engine loop has died (give a long-uptime relay time, or kill the task), the run never executes and the endpoint keeps returning `202` while nothing ever runs.

Contributor guide

Open the contributing guide

Research direction

Start at crates/buzz-relay/src/main.rs:658 and inspect how the spawned workflow engine task runs, exits, and reports failures. Reproduce the dead-task behavior with webhook, manual, and scheduled triggers, then trace how pending runs are created and consumed. Done should include a defined recovery path for engine failure, visibility into liveness, and no permanently pending accepted runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.