Rate-limit and dedup notifications for repeated failures
- Dominant language
- Rust
- Stars
- 3
- Forks
- 1
- Avg merge
- 4h 50m
- Merged PRs (30d)
- 1
Description
dotagent ships built-in notifier drivers (`desktop`, `slack`, `imessage`, `ntfy`, `pushover`) under `crates/dotagent-notify/`. They run in-process inside the daemon and fire on `attempt_failed` / `given_up` / `recovered` events.
If an agent enters a tight failure loop — e.g. a cron job that fails immediately every minute because of a missing credential — every single failure fires every configured notifier. A few real-world failure modes:
- An interval-2-minute agent failing for 6 hours = 180 notifications.
- A misconfigured Slack channel gets buried.
- iMessage/Pushover get rate-limited or silently dropped by the provider, so the *real* alert (when the user fixes things) gets lost.
There is no notifier-side dedup or rate-limiting today.
## Proposal
Add notification rate-limiting at the daemon level (not per-driver). Two behaviors:
1. **Dedup**: if the same `(agent, schedule, event_kind)` already fired a notification within a configurable window, suppress.
2. **Backoff**: after N consecutive failures with the same signature, back off exponentially (1m → 5m → 30m → 1h → cap) until either success or the user manually acks.
Defaults should be sensible out-of-the-box (per the "defaults sensatos" principle in [`CLAUDE.md`](CLAUDE.md)) — no config required, but overridable via a new `[notify_policy]` section in `config.toml` or per-agent.
A "recovered" notification should always go through, regardless of backoff state, so the operator knows the loop ended.
## Acceptance criteria
- [ ] An agent failing every minute for an hour produces ≪ 60 notifications (probably ~5-6 by default).
- [ ] The first failure still fires immediately. Backoff applies to *repeated* failures.
- [ ] Recovery always notifies, even mid-backoff.
- [ ] A user can override the policy globally (config) and per-agent (manifest).
- [ ] State (last-notified-at per signature) survives daemon restarts.
## Where to start
- `crates/dotagent-notify/src/lib.rs` — `fire_notifiers` is the single dispatch point.
- `crates/dotagent-state/` — for persisting "last notified" timestamps.
- [`docs/concepts/notifications.md`](docs/concepts/notifications.md) — current notifier semantics.
## Non-goals
- Per-driver rate limits (Slack's API caps etc.). This is application-level dedup; drivers can still get throttled by their own APIs and that is fine.
- Notification routing rules (severity-based fanout, on-call rotation). Future work.
Contributor guide
Research direction
Start with fire_notifiers in crates/dotagent-notify/src/lib.rs, then inspect crates/dotagent-state/ and docs/concepts/notifications.md for dispatch, persistence, and current semantics. Trace how daemon configuration and per-agent manifests are represented before defining the policy boundaries. Done means the acceptance criteria hold, including restart persistence, configurable global and per-agent behavior, immediate first failures, exponential backoff, and unconditional recovery notifications.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100