Scheduled (cron) workflows never fire on a hosted multi-tenant relay; manual trigger works
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
On a hosted multi-tenant Buzz deployment (`*.communities.buzz.xyz`), workflows
with `trigger: {on: schedule, cron: ...}` are accepted, listed, and execute
correctly via manual `kind:46020` trigger — but the cron scheduler never fires
them. No error is surfaced to the owner anywhere; the workflow simply does
nothing on schedule.
## Evidence (2026-08-30)
- A daily workflow (`cron: "0 9 * * *"`, created the previous evening, owner is
an active member of the target channel) did not fire at 09:00 UTC, and also
not at 16:00 UTC (ruling out a local-timezone interpretation). Manually
triggering the same workflow immediately posts its `send_message` step.
- A fresh probe workflow with `cron: "* * * * *"` (every minute) in a fresh
private channel produced **zero fires over 6.5 minutes** (~6 missed cron
instants). Same owner, same community, definition created via `buzz workflows
create` (relay-side `parse_yaml` accepted it).
- Relay NIP-11 reports `version 0.2.1`.
Client-side causes ruled out against current `main`:
- `WorkflowDef.enabled` defaults to `true` (`buzz-workflow/src/schema.rs`).
- 5-field cron is normalized for the `cron` crate by `normalize_cron`.
- The canonical definition JSON satisfies the scheduler's eligibility filter
(`definition->'trigger'->>'on' = 'schedule'`).
- `check_owner_authority` requires only active channel membership for a
`send_message`-only workflow — satisfied.
Since only relay logs can distinguish the remaining causes and community
operators/users have no access to them, filing here. Useful log greps:
`WorkflowEngine cron loop started`, `Cron tick: failed to load workflows`,
`Cron tick: invalid cron expression`, `Cron trigger fired`.
## A concrete bug regardless of the above: silent unbounded truncation
`list_all_enabled_workflows` (`crates/buzz-db/src/store/workflow.rs`) is the
scheduler's per-tick scan and is **platform-wide, ordered oldest-first, capped
at `LIST_MAX_LIMIT = 1000` rows**:
```sql
WHERE w.status = 'active'
AND w.enabled = TRUE
AND w.definition->'trigger'->>'on' = 'schedule'
AND c.archived_at IS NULL
ORDER BY w.created_at ASC
LIMIT $1 -- LIST_MAX_LIMIT = 1000
```
On any deployment where the total number of active scheduled workflows across
all communities exceeds 1000, every **newly created** scheduled workflow falls
past the limit and silently never fires — exactly the symptom above — with no
warning logged and no signal to the owner. Even if this isn't the cause here,
the query needs either pagination, removal of the cap, or at minimum a
`warn!` when `rows.len() == LIST_MAX_LIMIT`.
## Related observability gap
When the scheduler skips a workflow (parse failure, missing channel_id,
authority failure, claim errors), the only signal is a relay-side log line.
The owner-facing surfaces (workflow list, run history) show nothing. A
"last scheduled evaluation" status or an owner-visible error event would make
this class of failure diagnosable without relay access.
## Repro
1. On a multi-tenant deployment, create a workflow:
```yaml
name: scheduler-probe
trigger:
on: schedule
cron: "* * * * *"
steps:
- id: tick
action: send_message
text: scheduler tick
```
2. Wait 3+ minutes: no messages appear.
3. `buzz workflows trigger --workflow `: message appears immediately.
Contributor guide
Assessment
This issue has not been assessed yet.