buzz-acp: session-limit failures are dead-lettered ~25 min into a multi-hour limit — the message is lost, never reprocessed when quota returns
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
### Summary
When a turn fails with the provider's *session limit* error, `buzz-acp` retries it on the ordinary
budget and dead-letters it ~25 minutes later. The limit resets **hours** later, so the retry window
can never outlive it: the event is discarded before the agent could possibly have answered it, and
it is never reprocessed once quota returns. The message is lost for good.
The error text carries the reset instant, so the harness knows exactly how long it would have to
wait — it just doesn't use it.
### What I observed
Desktop 0.5.11 on macOS, managed agents against a Claude subscription. One agent, one DM channel,
three days (channel ids redacted, timestamps UTC):
| dead-lettered | events discarded |
|---|---|
| 2026-08-12 15:15:40Z | 2 |
| 2026-08-12 17:19:15Z | 1 |
| 2026-08-13 16:36:45Z | 1 |
| 2026-08-14 13:46:34Z | 1 |
Five of the owner's DMs to that agent, gone. Two more batches (a project channel and another
channel) were dead-lettered the same morning for the same reason.
Every one of the 10 retries preceding each dead-letter logged the same failure:
```
WARN buzz_acp: agent_returned (application error — pipe intact) agent=0 outcome="error"
configured_model=claude-fable-5[1m] pid=… error=Agent reported error (code -32603):
Internal error: You've hit your session limit · resets 1:50pm (America/Buenos_Aires)
WARN buzz_acp::queue: requeueing failed batch with backoff channel_id=… attempt=1 max=10 delay_secs=4.09
…
WARN buzz_acp::queue: requeueing failed batch with backoff channel_id=… attempt=10 max=10 delay_secs=246.46
ERROR buzz_acp::queue: dead-lettering batch after 10 retries — discarding 1 events channel_id=… attempt=11 events=1
```
First requeue 13:20:59Z → dead-letter 13:46:34Z: **25m35s of retrying**, against a limit that reset
at 16:50Z — **3h04m later**. Not one of the ten attempts had any chance of succeeding, and the batch
was discarded before the only moment it could have worked.
### Why this is structural, not bad luck
`crates/buzz-acp/src/queue.rs:29-36`:
```rust
pub(crate) const MAX_RETRIES: u32 = 10;
const BASE_RETRY_DELAY_SECS: u64 = 5; // doubled each attempt
const MAX_RETRY_DELAY_SECS: u64 = 300;
```
5 + 10 + 20 + 40 + 80 + 160 + 300 + 300 + 300 + 300 = **1515s ≈ 25 min**, and that is the ceiling,
not a typical case. A subscription session limit is measured in hours. So for this error class the
outcome is deterministic: retry budget exhausted, batch discarded, message never seen by the agent.
The user does get the `⚠️ I couldn't process the last request after multiple retries (…)` notice, so
this is not a silent drop — but "please re-send" arrives 25 minutes into a multi-hour outage, and if
the sender isn't watching (a DM overnight, an owner on another machine), the request is simply gone.
### Where it would fit
`handle_prompt_result` in `crates/buzz-acp/src/lib.rs` already classifies failures per error type
rather than treating them uniformly — `is_auth_error` at line 3956 dead-letters immediately, because
"the token won't self-repair between retries, so requeueing only wastes attempt slots". Session-limit
errors are the mirror image of that case: they *do* self-repair, at a known instant, just not inside
the retry budget. Today they fall through to the generic `queue.requeue(batch)` at line 3971.
### Proposal
1. **Preferred** — classify the limit error and park the batch until the reset instead of spending
retries on it. `EventQueue` already has a per-channel `retry_after`; the reset time is right there
in the error string. Parse it (fall back to a fixed backoff like 1h when it can't be parsed), set
`retry_after` to it, and don't count the attempt against `MAX_RETRIES`. The message is then
answered when quota returns, which is what the sender expects from an agent that is merely
throttled.
2. **Minimum** — if parking is unwanted, treat it like the auth class: dead-letter immediately with a
notice that names the reset time ("quota resets at 1:50pm — please re-send after that"), instead
of 25 minutes of silence followed by an undated "re-send".
Option 1 needs a per-adapter way to recognise the error, which is the same detection problem raised
in #5605 (there, quota exhaustion arrives as a *successful* turn whose text is the limit notice; here
it arrives as a `-32603` error). If the adapter surfaced quota exhaustion distinguishably, both
issues could key on it. Distinct from #5708 (missing notice on the panic path) and #4127 (dead
session never recycled) — here the session is fine and the notice is posted; the defect is that the
retry policy cannot span the outage it is retrying against.
Happy to send a PR for either option.
Contributor guide
Research direction
Start with crates/buzz-acp/src/queue.rs:29-36 to understand retry limits, backoff, and EventQueue's per-channel retry_after. Then inspect handle_prompt_result in crates/buzz-acp/src/lib.rs around lines 3956-3971, including is_auth_error and the generic requeue path. Done means session-limit failures remain available until quota returns, or are clearly reported with the reset time if parking is not used.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100