buzz-acp: 429s are retried with no backoff (median gap 0.0000s, 20-42/sec) — a transient rate limit becomes a self-sustaining storm
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
When the relay starts returning `429 Too Many Requests`, `buzz-acp` emits
retries with **no measurable delay between them** — hundreds per second. A
transient rate limit becomes a sustained, self-sustaining storm that the agent
never escapes, and which starves every other client on the same host.
## Measurement
Taken from a single agent's log (one `buzz-acp`, lazy pool, **zero** agent
child processes running):
```
429 events sampled : 175
inter-retry gap : min 0.0000s median 0.0000s max 0.65s
gaps under 100ms : 162 / 171
```
If there were exponential (or any) backoff, the gaps would trend upward over
the sample. They do not:
```
first 10 gaps, median : 0.0003s
last 10 gaps, median : 0.0000s
```
Aggregated per wall-clock second, the same log shows a **median of 20 and a
peak of 42** `429` responses per second, sustained.
## The part that makes it a storm, not just noise
While the agent was producing 20–42 rejected requests per second, a plain
`curl` to the same relay from the same host returned **8/8 × HTTP 200**. So the
relay was perfectly willing to serve this host at a normal request rate — the
agent's own retry volume was what kept it above the limit.
Earlier in the same session, with more agents running, `curl` got **0/10**. Stop
the agents and it returns to **10/10** immediately. The limit is reached by
client behaviour, and the retry loop is what holds it there.
## Two possible mechanisms — I can't distinguish them from outside
I want to be careful not to over-claim the cause:
1. **No backoff on retry** — each rejected request is retried immediately.
2. **High in-flight concurrency** — many parallel requests each get a 429 and
each logs, so the log rate reflects fan-out rather than retry rate.
The observable effect is the same and the fix probably overlaps (backoff plus a
concurrency cap on the publish path), but they're different bugs and worth
telling apart from inside the code. The log line is
`buzz_acp::relay: POST /events returned retriable HTTP 429 Too Many Requests`.
## Impact
This is the amplifier behind several other failures, including #5555 (an agent
that never joins a channel it has been added to) — the agent cannot make
progress because it is continuously rejected, and it cannot recover because
nothing slows it down.
It also degrades unrelated clients: with agents running, ordinary requests from
the same machine are rejected too, so a user's own desktop or CLI appears
broken while an agent is in this state.
## Suggested fix
- Exponential backoff with jitter on retriable statuses, and honour
`Retry-After` when present.
- A ceiling on concurrent in-flight publishes per agent.
- Log rate-limiting for this path — 175 identical lines in a few seconds buries
every other diagnostic in the terminal, which is how this went unnoticed
for the first hour.
## Context
macOS, Buzz Desktop 0.5.8, agent runtime `buzz-agent`, deployed through an
out-of-tree `buzz-backend-*` provider running agents in Orca worktrees.
Relay is self-hosted, Cloudflare-fronted.
Contributor guide
Assessment
This issue has not been assessed yet.