[Bug] buzz-acp: dead Claude Code session never recycled — events retried, dead-lettered, then bound channels go permanently silent
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
When the Claude Code process behind an ACP session crashes, buzz-acp keeps routing events to the dead session instead of recycling it. Every delivery fails with `-32603`, batches are retried with backoff and then dead-lettered (events discarded). After the final dead-letter the agent slot stops logging entirely, and every channel bound to it becomes a black hole: the agent still shows an online presence heartbeat but never responds. A message from a brand-new channel forced the harness to spawn a fresh session, which worked immediately — that is how we eventually recovered, ~5.5 hours later.
## Environment
- Buzz Desktop on Windows 11 Pro (10.0.26200), app id `xyz.block.buzz.app`
- buzz-acp harness, `agent_cmd=claude-agent-acp.cmd` (`@agentclientprotocol/claude-agent-acp` 0.64.0, bundled Node v24.18.0)
- Harness config from startup line: `idle_timeout=900s max_turn=7200s agents=10 subscribe=Mentions dedup=Queue meh=Steer permission_mode=bypassPermissions`
- Agent model: claude-fable-5
## Timeline (agent log `agents/logs/__.log`, 2026-08-01, UTC)
1. **07:05:35** — first failure while a turn was in flight:
```
WARN buzz_acp: agent_returned (application error — pipe intact) agent=0 outcome="error" pid=29144
error=Agent reported error (code -32603): Internal error: [ede_diagnostic] result_type=user last_content_type=n/a stop_reason=null
```
2. Immediately after, the adapter reports the child process is gone:
```
Session 02f491a6-ab1f-45d0-a184-2ebadb9141f7: Claude Agent process died: Claude Code process exited with code 3221226505
```
(3221226505 = `0xC0000409` STATUS_STACK_BUFFER_OVERRUN, a fail-fast abort.)
3. **07:13:18** — next delivery attempt returns:
```
Agent reported error (code -32603): Internal error: The Claude Agent process exited unexpectedly. Please start a new session.
```
4. **07:13 → 07:38** — the same batch is requeued with exponential backoff, attempts 1..10, every attempt hitting the same dead session. Then:
```
ERROR buzz_acp::queue: dead-lettering batch after 10 retries — discarding 3 events channel_id= attempt=11 events=3
```
5. **07:44 → 09:11** — the cycle repeats for newly arriving events; at **09:11:34** a second dead-letter discards 2 more events.
6. **09:11:34 → 12:47:00** — no queue/agent log lines at all. During this window the workspace owner sent multiple DMs and mentions to the agent (four messages over ~3 hours); none were delivered, nothing was logged, and no error was surfaced anywhere user-visible.
7. **12:47:00** — `INFO buzz_acp: membership notification: subscribing to new channel channel_id=` (a different user opened a brand-new DM with the agent) → the harness spawned a fresh session for it, which worked immediately and has been healthy since. The originally bound channels (owner DM, team channel) remained dead until the harness was restarted.
## Problems
1. **Dead session is never recycled.** The harness logs `Claude Agent process died`, and the adapter's own error text says "Please start a new session" — but deliveries keep being classified as `application error — pipe intact` and retried into the pinned dead session. The pipe to the *adapter* is intact; the session behind it is gone.
2. **Dead-lettering silently discards user events.** Real messages and mentions are dropped after 10 retries with no owner-visible notification and no redelivery once a healthy session exists.
3. **After the final dead-letter the slot goes fully silent.** Subsequent events for channels bound to that agent slot produce no queue activity at all — not even retry logs — so the failure also stops being observable.
4. **Net effect:** the agent looks online (presence heartbeat keeps publishing) while its owner's DM channel is a black hole for hours. The owner had to reach the agent through a second user's DM, which happened to force a fresh session.
## Expected behavior
- On `-32603` with a session-dead signal (`process died` / "Please start a new session"): tear down and respawn the ACP session (or open a new session) before retrying, instead of retrying into a dead session.
- Dead-lettered events should surface to the owner, or be redelivered once a healthy session for that channel exists.
- A dead/wedged agent slot should not keep channels permanently bound to it while presence continues to report the agent as online.
The underlying Claude Code crash (exit `0xC0000409` following the `ede_diagnostic` internal error) is an upstream agent issue; this report is about the harness's handling — a single agent-process crash should not take an agent's channels offline for hours.
Possibly related: #2961 (per-channel ACP session lifecycle on Windows — sessions never released).
Contributor guide
Research direction
Start by tracing the buzz_acp::queue retry and dead-letter path alongside the ACP adapter's session lifecycle, using the agent log sequence in this report as the failure trace. Compare the behavior with the per-channel session lifecycle discussion in issue #2961. Done means a dead ACP process is detected, affected channels recover through a new session, and failed events remain visible or are redelivered instead of becoming permanently silent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100