buzz-acp: hung in-flight turn has no watchdog — steer renews deadline, withheld mentions invisible until manual restart
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
When a buzz-acp in-flight turn's agent read loop dies (agent process alive but wedged), there is **no effective watchdog**: the turn stays "in-flight" for up to the renewed 2h deadline, a single successful steer **renews** that deadline, subsequent mentions get no steer ack and sit in the withheld table, and deadline expiry only recovers queue events — it never cancels the hung turn. The channel is effectively blocked until the agent instance is manually restarted, with no error surfaced anywhere.
## Observed incident (2026-08-05, Codex agent, Windows)
- 07:56:48 UTC — agent ACKs a task, starts investigating; research notes complete on disk at 08:02:15 UTC; **final reply never sent**.
- 08:03:54 — last buzz-acp log line; process tree (buzz-acp → codex-acp adapter → codex app-server → session kernel) all alive, CPU ≈ 0, no child processes, no `agent_returned`, no error, no restart (managed-agents.json `last_error` empty, `last_started_at` unchanged).
- 08:20:55 — a mention for the channel is steered: `non-cancelling steer ack received ... ack=Ok(Success) release_withheld=false drop_withheld=true` + `extending in-flight deadline by 7200s + 100s buffer`. The agent "received" the steer but produced nothing in the next 10+ minutes.
- 08:27:41 — next mention: **no steer ack at all** (read loop no longer selecting the steer arm). Message parked in `withheld_native_steer`.
- Only recovery: manual restart of the agent instance in Desktop.
## Mechanism (buzz-repo @ 7bee84d)
- `crates/buzz-acp/src/lib.rs:2522-2627` — on `SteerAck::Success` the withheld event is dropped and `extend_in_flight_deadline(channel_id, max_turn_duration)` renews the in-flight deadline (7200s).
- `crates/buzz-acp/src/lib.rs:1147-1164, 2256` + `crates/buzz-acp/src/pool.rs:300-421` — steer is a capacity-1 per-turn channel consumed by the prompt read loop (`acp.rs:1280-1347`). If the read loop is wedged (blocked in a model read or synchronous call), the steer request never gets an ack; there is no timeout on the ack itself.
- `crates/buzz-acp/src/queue.rs:155-166, 280-287, 576-580, 758-771` — withheld steer events are only recovered on in-flight-deadline expiry; expiry does not cancel the turn or signal the agent. A `Err(_)` steer ack would fire the cancel+merge fallback (`signal_fallback`), but a **missing** ack is never treated as failure.
- No heartbeat exists between buzz-acp and the agent while a turn is running; `agent_returned` never fires for the wedged turn.
## Impact
- Channel blocked for up to 2h per renewed deadline; further mentions are silently invisible (no ack, no error).
- Operator-facing symptom: agent process "alive" (green), zero output, no logs — indistinguishable from idle without deep inspection.
- Same symptom class as the Hermes hang on Windows (process alive, CPU ≈ 0, no timeout fired) — likely a family of "wedged tool/model call inside a turn" cases.
## Expected behavior
- A watchdog should detect a non-responsive in-flight turn (no steer ack and/or no progress for a bounded window — do not let steer success renew the deadline indefinitely) and:
1. cancel the turn via the existing cancel+merge fallback path (`ControlSignal::Steer`, `signal_fallback` semantics in `lib.rs:2522-2627`),
2. surface an error/status to the Desktop UI (the agent currently shows healthy),
3. recover the withheld events to the queue for redelivery.
- Consider a per-turn progress heartbeat between buzz-acp and the agent, with a hard cap that cannot be extended by steers alone.
## Related
- #4477 (transient network outage paralyzes agent; passive recovery)
- #4098 (Hermes ACP prompts hang until 60s timeout on Windows)
- #3648 (retry channel turns that leave a mention unanswered — closed)
- #2554 (name the hung RPC method in request-timeout errors)
Contributor guide
Assessment
This issue has not been assessed yet.