buzz-agent/buzz-acp: transient network outage paralyzes agent for 12+ min; retries noisy and recovery passive
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
A brief network outage (DNS failure lasting ~30s) paralyzed a managed `buzz-agent` for **12+ minutes**, and the user-visible fallout ("turn error" states in the desktop UI, silent message queueing) persisted long after the network recovered. The harness eventually self-healed via requeue, but the recovery path is slow, opaque, and noisy. Observed on a production multi-agent deployment.
## Observed timeline (all times UTC, real logs)
| Time | Event |
|---|---|
| 04:00:03 | User DMs the agent; message steered into an in-flight turn, deadline extended +7200s |
| (network) | macOS DNS for the LLM gateway domain breaks; route black-holes |
| 04:02:49 | `llm: transport error, retrying attempt=1 max_attempts=3` — **~2.5 min after the request started hanging** |
| 04:06:49 | attempt=2 — another ~4 min hang |
| 04:10:50 | `cumulative stall 721s across 3 attempts` → `session_prompt error code -32000` |
| 04:10:50 | `queue: requeueing failed batch with backoff attempt=1 max=10` |
| 04:12 | Network confirmed healthy again (gateway reachable, 200 OK) |
| 04:14:56 | Requeued batch hits transport error **again** (stale connection state / still-settling routes) |
| 04:19:46 | `WebSocket error: Connection reset without closing handshake` |
| 04:26+ | Agent finally drains the backlog and replies to all queued DMs |
## Problems
**1. Each failed LLM attempt hangs minutes before erroring.**
`crates/buzz-agent/src/llm.rs` retries with sensible backoff (`MAX_RETRIES=3`, 500ms–8s jitter), but the transport error itself takes ~4 minutes to surface per attempt (black-holed route → long TCP/TLS timeout). So a 30-second DNS blip costs 3 × ~4min ≈ **12 minutes of turn paralysis**. There is no aggressive per-attempt connect/request timeout for the chat-completions call.
**2. Queued messages pile up behind the doomed turn.**
New messages arriving during the stall become non-cancelling steers that only extend the hard deadline (`extending in-flight deadline by 7200s + 100s buffer`) — from the user's perspective the agent simply ignores them.
**3. The failure/recovery is user-visible and noisy.**
`agent_returned outcome="error"` propagates to observer frames, so the desktop shows "turn error" in the conversation while the queue is *still retrying* (requeue max=10). The user concluded "DM is broken" and started restarting the agent — which actually made diagnosis harder. Transient transport failures that will be retried should not surface as terminal-looking errors.
**4. No proactive recovery trigger.**
After the network heals, nothing probes the LLM endpoint to fast-retry queued batches — recovery waits for whatever backoff/deadline timer fires next (in our case ~10 extra minutes).
## Proposals
1. **Bounded per-attempt timeout** for the LLM HTTP call (e.g. 30–60s, configurable): transport failures should surface in seconds, not minutes.
2. **Circuit breaker for transport errors**: after N consecutive transport failures, pause new turns, probe the endpoint in the background, and resume + drain the queue immediately when it recovers — instead of riding out long backoff timers.
3. **Suppress user-visible error state while retries remain**: only surface "turn error" to observers when the requeue budget is actually exhausted; mark intermediate failures as transient (e.g. a "retrying" state distinct from "error").
4. Optional: **fail-fast for queued steers** — if the in-flight turn is already doomed (LLM unreachable), cancel it at the next retry boundary and re-batch the steered messages, rather than extending a 7200s deadline.
Happy to implement (1) and (2) if the direction sounds right — we have the repro environment and logs.
## Environment
- `buzz-acp` + built-in `buzz-agent` (OpenAI-compatible provider, GLM), `subscribe=all`, `turn_timeout=320s`, `max_turn=7200s`
- macOS desktop (Tauri managed agents), relay behind an LB over WSS
- Trigger: client-side DNS outage of the LLM gateway domain (~30s), Clash TUN fake-ip involved
Contributor guide
Research direction
Start with crates/buzz-agent/src/llm.rs and trace the retry, requeue, and observer-frame paths described in the report. Determine how transport timeouts, queued turns, and in-flight steers interact before choosing the implementation scope. Done should include bounded transport attempts, faster recovery after endpoint availability returns, and non-terminal status while retries remain.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, tauri
- Domain
- backend, desktop, networking
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100