avelino / avelino/dotagent

Rate-limit and dedup notifications for repeated failures

Open
#7 0 comments 0 reactions 0 assignees View on GitHub
area: notify type: feature
Dominant language
Rust
Stars
3
Forks
1
Avg merge
4h 50m
Merged PRs (30d)
1

Description

dotagent ships built-in notifier drivers (`desktop`, `slack`, `imessage`, `ntfy`, `pushover`) under `crates/dotagent-notify/`. They run in-process inside the daemon and fire on `attempt_failed` / `given_up` / `recovered` events.

If an agent enters a tight failure loop — e.g. a cron job that fails immediately every minute because of a missing credential — every single failure fires every configured notifier. A few real-world failure modes:

- An interval-2-minute agent failing for 6 hours = 180 notifications.
- A misconfigured Slack channel gets buried.
- iMessage/Pushover get rate-limited or silently dropped by the provider, so the *real* alert (when the user fixes things) gets lost.

There is no notifier-side dedup or rate-limiting today.

## Proposal

Add notification rate-limiting at the daemon level (not per-driver). Two behaviors:

1. **Dedup**: if the same `(agent, schedule, event_kind)` already fired a notification within a configurable window, suppress.
2. **Backoff**: after N consecutive failures with the same signature, back off exponentially (1m → 5m → 30m → 1h → cap) until either success or the user manually acks.

Defaults should be sensible out-of-the-box (per the "defaults sensatos" principle in [`CLAUDE.md`](CLAUDE.md)) — no config required, but overridable via a new `[notify_policy]` section in `config.toml` or per-agent.

A "recovered" notification should always go through, regardless of backoff state, so the operator knows the loop ended.

## Acceptance criteria

- [ ] An agent failing every minute for an hour produces ≪ 60 notifications (probably ~5-6 by default).
- [ ] The first failure still fires immediately. Backoff applies to *repeated* failures.
- [ ] Recovery always notifies, even mid-backoff.
- [ ] A user can override the policy globally (config) and per-agent (manifest).
- [ ] State (last-notified-at per signature) survives daemon restarts.

## Where to start

- `crates/dotagent-notify/src/lib.rs` — `fire_notifiers` is the single dispatch point.
- `crates/dotagent-state/` — for persisting "last notified" timestamps.
- [`docs/concepts/notifications.md`](docs/concepts/notifications.md) — current notifier semantics.

## Non-goals

- Per-driver rate limits (Slack's API caps etc.). This is application-level dedup; drivers can still get throttled by their own APIs and that is fine.
- Notification routing rules (severity-based fanout, on-call rotation). Future work.

Contributor guide

Open the contributing guide

Research direction

Start with fire_notifiers in crates/dotagent-notify/src/lib.rs, then inspect crates/dotagent-state/ and docs/concepts/notifications.md for dispatch, persistence, and current semantics. Trace how daemon configuration and per-agent manifests are represented before defining the policy boundaries. Done means the acceptance criteria hold, including restart persistence, configurable global and per-agent behavior, immediate first failures, exponential backoff, and unconditional recovery notifications.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.