block / block/buzz

buzz-acp: zombie in-flight turn after reconnect + rate-limit; agent acks mentions but never responds

Open
#4,667 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

`buzz-acp` gets stuck with a zombie in-flight turn after a WebSocket disconnect + relay rate-limit. The agent process stays online and heartbeat-keepalive, incoming @mentions are accepted with `ack=Ok(Success)` (steer path), but the agent **never produces any output** again — every subsequent mention is steered into a dead turn and silently dropped.

## Observed Behavior

1. Agent (claude-agent-acp runtime, 10 workers, `steering_supported=true`) initialized fine and worked normally for hours.
2. Relay connection dropped (`Connection reset without closing handshake`), then 4 autonomous reconnect attempts (several `502 Bad Gateway`), finally reconnect succeeded.
3. Immediately after reconnect, relay sent `NOTICE: rate-limited: quota exceeded`, and the relay observer logged:
```
relay observer publisher lagged dropped=108
relay observer publisher lagged dropped=2017
```
→ **2017+ events dropped during the reconnect/backpressure window.**
4. After this window, every new @mention is handled via the non-cancelling steer path and acked:
```
steer success: renewed hard deadline (7200s from now)
non-cancelling steer ack received channel= event_id= ack=Ok(Success) release_withheld=false drop_withheld=true signal_fallback=false
extending in-flight deadline by 7200s + 100s buffer channel_id=
```
5. No turn completion, no tool calls, no message publishing ever happens again. The claude worker process shows near-zero CPU growth (zombie/idle). `presence` stays `online`.

## Expected Behavior

After a relay disconnect / rate-limit / event-drop, an in-flight turn should be recovered or expired so that the next @mention starts a fresh turn. The agent should keep responding to mentions rather than silently dead-ending.

## Root-cause hypothesis

The in-flight turn state machine believes a turn is still running (`extend in-flight deadline` keeps firing), so new mentions are treated as mid-turn steer injection into a turn whose claude worker is no longer producing output. The turn is never reclaimed — the deadline just keeps getting renewed 7200s at a time, so the normal recovery path (deadline expiry → recover withheld → cancel+merge) never triggers.

The `observer publisher lagged dropped=2017` during reconnect is the likely trigger: critical turn lifecycle events (completion/abort) were among the dropped events, leaving the queue with an orphaned in-flight turn.

## Environment

- Platform: Windows 11 (win32)
- `buzz-acp` via Buzz Desktop (agent harness), agent runtime `@agentclientprotocol/claude-agent-acp` 0.64.2 (`agent_command_override=claude-agent-acp`)
- Relay: self-hosted Nostr relay (`wss://buzz.tonyooo.xyz`), rate-limited (`quota exceeded`) during the incident
- Config: `subscribe=Mentions`, `meh=Steer`, `permission_mode=bypassPermissions`, `respond_to=allowlist`, `max_turn=7200s`

## Suggested fix directions

- After reconnect / when `observer lagged ... dropped` is observed, expire orphaned in-flight turns (or run the same recovery path as deadline expiry) so withheld mentions are re-dispatched as fresh turns.
- Consider a max consecutive-steer-without-output guard: if a turn is being steered repeatedly but produces nothing and its worker is idle, reclaim it.
- Emit a WARN/ERROR log when a mention is steered into a turn that has not produced output for a long time, to make this diagnosable from logs.

## Log excerpts (sanitized)

```
2026-08-03T22:28:49Z WARN buzz_acp::relay: WebSocket error: Connection reset without closing handshake
2026-08-03T22:28:49Z WARN buzz_acp::relay: autonomous reconnect attempt 1/5 failed: HTTP error: 502 Bad Gateway
2026-08-03T22:28:56Z INFO buzz_acp::relay: autonomous reconnect succeeded (attempt 4)
2026-08-03T22:29:36Z WARN buzz_acp: relay observer publisher lagged dropped=108
2026-08-03T22:29:46Z WARN buzz_acp: relay observer publisher lagged dropped=2017
2026-08-03T22:29:46Z WARN buzz_acp::relay: relay NOTICE: rate-limited: quota exceeded; retry in 10s
# hours later, a new mention:
2026-08-04T04:37:35Z INFO buzz_acp::acp: steer success: renewed hard deadline (7200s from now)
2026-08-04T04:37:35Z INFO buzz_acp: non-cancelling steer ack received channel=... event_id=... ack=Ok(Success) release_withheld=false drop_withheld=true signal_fallback=false
2026-08-04T04:37:35Z INFO buzz_acp::queue: extending in-flight deadline by 7200s + 100s buffer
# ...no further log lines, ever. Agent never responds again.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.