buzz-acp goes permanently silent after a burst of relay rate-limit NOTICEs — no crash, no further logs, restart required
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
**Environment:**
- Buzz.app 0.5.2 (macOS), `buzz-acp` / `@agentclientprotocol/claude-agent-acp` 0.63.0
- Relay: self-hosted `ghcr.io/block/buzz:main` (commit `bb34bc4`, 2026-07-31), no rate-limit env vars set at the time of the incident (relay defaults: `BUZZ_RATE_LIMIT_HUMAN_WS_EVENTS_PER_SEC=10`, fixed 5s window)
**Description:**
After a `buzz-acp` reconnect/resubscribe burst (e.g. right after an app restart, with multiple agent personas reconnecting near-simultaneously), the relay's fixed-window WS-event rate limiter (`BUZZ_RATE_LIMIT_HUMAN_WS_EVENTS_PER_SEC`, applied to agent connections too per `connection.rs`) frequently rejects with:
```
relay NOTICE: rate-limited: quota exceeded; retry in Ns
```
handled in `crates/buzz-acp/src/relay.rs` (`RelayMessage::Notice`), logged as:
```
WARN buzz_acp::relay: rate-limit gate armed via NOTICE until ~N.Ns from now
```
In most cases (~14/16 log samples we collected) the client recovers cleanly after the window. In a minority of cases, after several of these NOTICE/gate-arm lines fire in quick succession, the process goes **fully silent forever** — literally zero further log lines of any kind (not even at DEBUG), while remaining "online" in the UI and never processing another message. Only a full `Buzz.app` quit+relaunch recovers it.
We initially suspected a race in the gate re-arming (`set_rate_limit_gate`/`check_rate_gate`), but on reading the actual source, that logic is a cooperatively-polled `Option` using `max()` to extend the deadline — not a droppable timer task — and there are dedicated unit tests for exactly this "NOTICE arrives while a retry is pending" scenario. So the gate itself looks correctly defended. We could not identify the actual hang point from the client-visible log alone; our best remaining guess is something downstream of the 30s ping-tick-triggered reconnect path (`try_autonomous_reconnect` / `wait_for_reconnect`, not fully traced by us) blocking without a timeout, past where this module's own logging would fire — but we have not confirmed this.
We also checked our own setup against #2641 ("agent shows online but never responds", root-caused there to missing channel membership) — that doesn't match our case; our failing session had successfully subscribed to a channel just before going silent. Separately, we did notice the channel count discovered on restart dropped from 2 to 1 between two consecutive starts of the same agent identity, which may be an unrelated membership-resolution issue worth a look.
**Related internal note:** `docs/welcome-kickoff-silent-failures.md` documents a similar-but-distinct 2026-07-17 incident (a 42KB log of rapid rate-limit retries causing cascading send failures) with an open TODO ("worth a separate look at buzz-acp publish backoff") — that's a *loud* failure mode (lots of logging) whereas ours is *silent* (zero logging), so likely a different bug sharing the same trigger condition (rate-limit NOTICE burst).
**Impact:** Agents silently stop responding for hours with no visible error state, discoverable only via local log inspection. Recurred multiple times in one day for us.
**What we've done as a workaround:** raised `BUZZ_RATE_LIMIT_HUMAN_WS_EVENTS_PER_SEC` and the agent-tier message-per-minute limits on our self-hosted relay to reduce trigger frequency — this should help but doesn't address the underlying silent hang.
**Ask:** if anyone on the team can point at what happens after the ping-tick fires and a reconnect is attempted while the rate-limit gate is armed, that would help narrow down where the silent hang actually occurs.
Contributor guide
Research direction
Start in crates/buzz-acp/src/relay.rs and connection.rs, tracing RelayMessage::Notice, set_rate_limit_gate, check_rate_gate, and the ping-tick reconnect path through try_autonomous_reconnect and wait_for_reconnect. Review the existing unit tests for a NOTICE arriving while a retry is pending, then reproduce the reconnect burst with logging around the downstream wait. Done means the silent hang is localized and covered by a regression test that confirms the client either reconnects or reports an error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 48/100