block / block/buzz

buzz-acp goes permanently silent after a burst of relay rate-limit NOTICEs — no crash, no further logs, restart required

Open
#3,969 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

**Environment:**
- Buzz.app 0.5.2 (macOS), `buzz-acp` / `@agentclientprotocol/claude-agent-acp` 0.63.0
- Relay: self-hosted `ghcr.io/block/buzz:main` (commit `bb34bc4`, 2026-07-31), no rate-limit env vars set at the time of the incident (relay defaults: `BUZZ_RATE_LIMIT_HUMAN_WS_EVENTS_PER_SEC=10`, fixed 5s window)

**Description:**

After a `buzz-acp` reconnect/resubscribe burst (e.g. right after an app restart, with multiple agent personas reconnecting near-simultaneously), the relay's fixed-window WS-event rate limiter (`BUZZ_RATE_LIMIT_HUMAN_WS_EVENTS_PER_SEC`, applied to agent connections too per `connection.rs`) frequently rejects with:
```
relay NOTICE: rate-limited: quota exceeded; retry in Ns
```
handled in `crates/buzz-acp/src/relay.rs` (`RelayMessage::Notice`), logged as:
```
WARN buzz_acp::relay: rate-limit gate armed via NOTICE until ~N.Ns from now
```
In most cases (~14/16 log samples we collected) the client recovers cleanly after the window. In a minority of cases, after several of these NOTICE/gate-arm lines fire in quick succession, the process goes **fully silent forever** — literally zero further log lines of any kind (not even at DEBUG), while remaining "online" in the UI and never processing another message. Only a full `Buzz.app` quit+relaunch recovers it.

We initially suspected a race in the gate re-arming (`set_rate_limit_gate`/`check_rate_gate`), but on reading the actual source, that logic is a cooperatively-polled `Option` using `max()` to extend the deadline — not a droppable timer task — and there are dedicated unit tests for exactly this "NOTICE arrives while a retry is pending" scenario. So the gate itself looks correctly defended. We could not identify the actual hang point from the client-visible log alone; our best remaining guess is something downstream of the 30s ping-tick-triggered reconnect path (`try_autonomous_reconnect` / `wait_for_reconnect`, not fully traced by us) blocking without a timeout, past where this module's own logging would fire — but we have not confirmed this.

We also checked our own setup against #2641 ("agent shows online but never responds", root-caused there to missing channel membership) — that doesn't match our case; our failing session had successfully subscribed to a channel just before going silent. Separately, we did notice the channel count discovered on restart dropped from 2 to 1 between two consecutive starts of the same agent identity, which may be an unrelated membership-resolution issue worth a look.

**Related internal note:** `docs/welcome-kickoff-silent-failures.md` documents a similar-but-distinct 2026-07-17 incident (a 42KB log of rapid rate-limit retries causing cascading send failures) with an open TODO ("worth a separate look at buzz-acp publish backoff") — that's a *loud* failure mode (lots of logging) whereas ours is *silent* (zero logging), so likely a different bug sharing the same trigger condition (rate-limit NOTICE burst).

**Impact:** Agents silently stop responding for hours with no visible error state, discoverable only via local log inspection. Recurred multiple times in one day for us.

**What we've done as a workaround:** raised `BUZZ_RATE_LIMIT_HUMAN_WS_EVENTS_PER_SEC` and the agent-tier message-per-minute limits on our self-hosted relay to reduce trigger frequency — this should help but doesn't address the underlying silent hang.

**Ask:** if anyone on the team can point at what happens after the ping-tick fires and a reconnect is attempted while the rate-limit gate is armed, that would help narrow down where the silent hang actually occurs.

Contributor guide

Open the contributing guide

Research direction

Start in crates/buzz-acp/src/relay.rs and connection.rs, tracing RelayMessage::Notice, set_rate_limit_gate, check_rate_gate, and the ping-tick reconnect path through try_autonomous_reconnect and wait_for_reconnect. Review the existing unit tests for a NOTICE arriving while a retry is pending, then reproduce the reconnect burst with logging around the downstream wait. Done means the silent hang is localized and covered by a regression test that confirms the client either reconnects or reports an error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.