block / block/buzz

buzz-agent/buzz-acp: transient network outage paralyzes agent for 12+ min; retries noisy and recovery passive

Open
#4,477 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

A brief network outage (DNS failure lasting ~30s) paralyzed a managed `buzz-agent` for **12+ minutes**, and the user-visible fallout ("turn error" states in the desktop UI, silent message queueing) persisted long after the network recovered. The harness eventually self-healed via requeue, but the recovery path is slow, opaque, and noisy. Observed on a production multi-agent deployment.

## Observed timeline (all times UTC, real logs)

| Time | Event |
|---|---|
| 04:00:03 | User DMs the agent; message steered into an in-flight turn, deadline extended +7200s |
| (network) | macOS DNS for the LLM gateway domain breaks; route black-holes |
| 04:02:49 | `llm: transport error, retrying attempt=1 max_attempts=3` — **~2.5 min after the request started hanging** |
| 04:06:49 | attempt=2 — another ~4 min hang |
| 04:10:50 | `cumulative stall 721s across 3 attempts` → `session_prompt error code -32000` |
| 04:10:50 | `queue: requeueing failed batch with backoff attempt=1 max=10` |
| 04:12 | Network confirmed healthy again (gateway reachable, 200 OK) |
| 04:14:56 | Requeued batch hits transport error **again** (stale connection state / still-settling routes) |
| 04:19:46 | `WebSocket error: Connection reset without closing handshake` |
| 04:26+ | Agent finally drains the backlog and replies to all queued DMs |

## Problems

**1. Each failed LLM attempt hangs minutes before erroring.**
`crates/buzz-agent/src/llm.rs` retries with sensible backoff (`MAX_RETRIES=3`, 500ms–8s jitter), but the transport error itself takes ~4 minutes to surface per attempt (black-holed route → long TCP/TLS timeout). So a 30-second DNS blip costs 3 × ~4min ≈ **12 minutes of turn paralysis**. There is no aggressive per-attempt connect/request timeout for the chat-completions call.

**2. Queued messages pile up behind the doomed turn.**
New messages arriving during the stall become non-cancelling steers that only extend the hard deadline (`extending in-flight deadline by 7200s + 100s buffer`) — from the user's perspective the agent simply ignores them.

**3. The failure/recovery is user-visible and noisy.**
`agent_returned outcome="error"` propagates to observer frames, so the desktop shows "turn error" in the conversation while the queue is *still retrying* (requeue max=10). The user concluded "DM is broken" and started restarting the agent — which actually made diagnosis harder. Transient transport failures that will be retried should not surface as terminal-looking errors.

**4. No proactive recovery trigger.**
After the network heals, nothing probes the LLM endpoint to fast-retry queued batches — recovery waits for whatever backoff/deadline timer fires next (in our case ~10 extra minutes).

## Proposals

1. **Bounded per-attempt timeout** for the LLM HTTP call (e.g. 30–60s, configurable): transport failures should surface in seconds, not minutes.
2. **Circuit breaker for transport errors**: after N consecutive transport failures, pause new turns, probe the endpoint in the background, and resume + drain the queue immediately when it recovers — instead of riding out long backoff timers.
3. **Suppress user-visible error state while retries remain**: only surface "turn error" to observers when the requeue budget is actually exhausted; mark intermediate failures as transient (e.g. a "retrying" state distinct from "error").
4. Optional: **fail-fast for queued steers** — if the in-flight turn is already doomed (LLM unreachable), cancel it at the next retry boundary and re-batch the steered messages, rather than extending a 7200s deadline.

Happy to implement (1) and (2) if the direction sounds right — we have the repro environment and logs.

## Environment

- `buzz-acp` + built-in `buzz-agent` (OpenAI-compatible provider, GLM), `subscribe=all`, `turn_timeout=320s`, `max_turn=7200s`
- macOS desktop (Tauri managed agents), relay behind an LB over WSS
- Trigger: client-side DNS outage of the LLM gateway domain (~30s), Clash TUN fake-ip involved

Contributor guide

Open the contributing guide

Research direction

Start with crates/buzz-agent/src/llm.rs and trace the retry, requeue, and observer-frame paths described in the report. Determine how transport timeouts, queued turns, and in-flight steers interact before choosing the implementation scope. Done should include bounded transport attempts, faster recovery after endpoint availability returns, and non-terminal status while retries remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, tauri
Domain
backend, desktop, networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.