block / block/buzz

buzz-acp: an agent added to a channel during relay rate-limiting never joins it and never recovers — no membership reconciliation after queue overflow

Open
#5,555 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

An agent added to a channel while the relay is rate-limiting never joins it,
and **never recovers** — it stays at zero channel subscriptions indefinitely,
long after the relay is healthy again. There is no periodic reconciliation of
channel membership, so a single overflow window strands the agent permanently.

## What I observed

A freshly deployed managed agent started, connected, and reported:

```
INFO buzz_acp: connected to relay at wss://…
INFO buzz_acp: discovered 0 channel(s)
```

Zero channels is correct at that moment — it had not been added to one yet.

I then added it to a channel through Buzz Desktop (agent profile → Channels →
Add to channel, role `bot`). The dialog succeeded.

Meanwhile the relay began rate-limiting this host hard. Over the next few
minutes the agent's log accumulated:

| Log line | Count |
|---|---|
| `POST /events returned retriable HTTP 429 Too Many Requests` | 172 |
| `queue depth cap reached — dropped oldest event` | 78 |
| `POST /events network error: error sending request` | 12 |

The rate limiting was **not** specific to this agent — a plain
`curl https:///` from the same host returned `HTTP 429` at the same
time, so it was an IP-level limit affecting every client on the machine.

The relay recovered (`curl` → `HTTP 200`). **Ninety minutes later the agent
still had zero channel subscriptions** — no `subscribed to channel …` line ever
appeared, and it could not be mentioned.

## Why this is more than "the relay was busy"

The transient failure is expected and fine. The problem is that the recovery
path doesn't exist:

- `buzz-acp` subscribes to membership notifications once at startup and reacts
to them as they arrive.
- Channel discovery runs **once**, at startup.
- When the inbound queue overflows (`queue depth cap reached`), whatever was
dropped is simply gone.
- Nothing re-runs discovery or re-reconciles membership afterwards — not on
queue overflow, not on reconnect, not on a timer.

So the window in which a membership notification can be lost is small, but the
consequence is permanent and silent: an agent that looks completely healthy
(process alive, relay connected, green dot in the UI) and is simply not in any
channel.

## Impact

Worse for **provider-backed agents**, which cannot be restarted. Protocol v1
has no `undeploy`, Stop and Start are no-ops for them (see #4605), so the
documented recovery — restart the agent — is unavailable. The only remedy is
delete and redeploy.

## Suggested fix

Any of these would close it:

1. **Re-run channel discovery after a queue overflow.** The `queue depth cap
reached` path already knows something was dropped; treat that as "my view of
membership may be stale".
2. **Periodic membership reconciliation** — cheap, and self-heals every cause of
drift, not just this one.
3. **Re-discover on reconnect**, since a reconnect already implies a gap.

## Related

- #4564 — also about membership notifications, but the opposite direction: the
notification *is* processed and the replay floor drops older messages. Here
the subscription never happens at all.
- #4605 — why the usual "restart it" workaround is unavailable for
provider-backed agents.

Observed on Buzz Desktop 0.5.8 (macOS), agent runtime `buzz-agent`, deployed
through a third-party `buzz-backend-*` provider.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by locating the startup channel discovery, membership-notification handling, inbound queue-overflow, and reconnect paths described in the issue. Done means an agent whose membership notification is lost eventually discovers and joins its assigned channel after the relay recovers, with coverage for the overflow scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.