block / block/buzz

[Bug] Per-channel ACP sessions are never released: on Windows each live session pins a full agent process tree (~33 procs / ~2 GB), exhausting the commit limit until Windows kills the WebView2 host and the desktop silently renders black

Open
#2,961 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

On Windows, a Buzz Desktop instance running 8 managed agents (each with `parallelism: 1`) grew to **~140 agent-owned OS processes and ~36 GB of commit charge over ~6 hours**, exhausted the system commit limit, and the OS reclaimed memory by terminating **Buzz's own WebView2 host process**. The Tauri window survived with no renderer, so the app rendered as a solid black window.

The failure is silent: `buzz-desktop.exe` keeps running, the process stays responsive, and **nothing is written to the app log, the agent logs, or stdout/stderr**. There is no in-app indication that the UI is dead or why.

The growth is not an orphan/zombie leak — the process tree is well-formed and a graceful quit reclaims everything. The driver is that **ACP sessions are keyed per channel and are never released for the lifetime of the worker**, and with `claude-agent-acp` every live session materialises as a full agent CLI process tree. Session count therefore grows toward `agents × channels`, and memory grows linearly with it, with no cap, no idle eviction, and no visibility.

This is a different mechanism from #2631 (that one is `DEFAULT_AGENT_PARALLELISM = 24` + eager pool init on manual start). Here parallelism is **1** on every running agent and the pool starts lazily via the restore path — the multiplication comes from per-channel sessions instead. The two compound.

## Environment

- Buzz Desktop **0.4.26**, stock official build, self-hosted relay at `127.0.0.1:3000`
- Windows 11 Pro 26200, 32 GB RAM, commit limit 73 GB (system-managed pagefile)
- WebView2 Runtime 150.0.4078.83
- Agent runtime: `@agentclientprotocol/claude-agent-acp` (system npm), `agent_command: "claude-agent-acp"`
- 8 managed agents with `start_on_app_launch: true`, all `parallelism: 1`, `idle_timeout_seconds: null`
- Each agent subscribes to 2–6 channels (`discovered N channel(s)` in the agent logs); ~30 (agent, channel) pairs in total

**Binary provenance — verified, no local patches involved.** Every executable in the install directory is byte-identical to the ones inside the official `Buzz_0.4.26_x64-setup_alpha-unsigned.exe` release asset. I extracted the published installer and compared SHA-256:

| file | SHA-256 (first 16) | official release asset | installed |
|---|---|---|---|
| `buzz.exe` | `639EAA825D222C1E` | ✔ | ✔ |
| `buzz-acp.exe` | `C42DDE0E463048AC` | ✔ | ✔ |
| `buzz-agent.exe` | `CE05F3C566273D6F` | ✔ | ✔ |
| `buzz-desktop.exe` | `4E017BCD1E19B24B` | ✔ | ✔ |
| `buzz-dev-mcp.exe` | `9AC3124F3B5699FD` | ✔ | ✔ |
| `git-credential-nostr.exe` | `AA3B3458FA50DA0B` | ✔ | ✔ |

All six match.

**However — and this is a correction to an earlier version of this report — the `buzz-acp` binary that was actually *running* was not the installed one.** `resolve_command()` puts `/target/release` and `/target/debug` at the front of `command_search_dirs()` (`desktop/src-tauri/src/managed_agents/discovery.rs`), ahead of the install directory. My launcher script `Set-Location`'d into a `block/buzz` source checkout before starting `buzz-desktop`, so every agent resolved `buzz-acp` to a locally built debug binary (built 2026-07-24 from a branch based on an older `main`) instead of `E:\Programs\Buzz\buzz-acp.exe`. All 9 live `buzz-acp.exe` processes had `ExecutablePath = ...\target\debug\buzz-acp.exe`.

What that does and does not change:

- **Does not change the mechanism.** The session-management code is identical between the built branch and `v0.4.26`: `git diff v0.4.26 HEAD -- crates/buzz-acp/src/pool.rs` shows no change to any line touching `sessions`, `invalidate`, `session_new`, or `max_turns`. The `channel → session` map with no eviction is the same code in both.
- **Does not change the measurements of cost.** The 33-process / 2.06 GB figure is the `claude` CLI process tree spawned by `claude-agent-acp`, downstream of `buzz-acp` entirely.
- **Does mean the harness binary was not stock.** I have fixed the launcher to pass an explicit `WorkingDirectory` so the installed binary wins, and I am re-running the measurement on the stock `buzz-acp`. I will post the stock numbers as a follow-up comment. If they differ from the above in any way, I will say so.

I am flagging this rather than quietly re-measuring first, because a reviewer should not have to take my word for which binary produced a number.

## What happened

| Time (JST, 2026-07-26) | Event |
|---|---|
| 11:47 | `buzz-desktop.exe` started, 8 agents auto-started |
| 17:45:38 | System log `Id 26` (Application Popup, "virtual memory low") and `Id 2004` (`Microsoft-Windows-Resource-Exhaustion-Detector`). Commit charge had reached **65.1 GB / 73.0 GB** |
| 17:45:47–49 | `%LOCALAPPDATA%\xyz.block.buzz.app\EBWebView\Crashpad\` updated (`edge_shutdown_crash.txt`, `metadata`). **No minidump written** — consistent with external termination, not a self-crash |
| ~17:46 onward | Window renders solid black. `buzz-desktop.exe` still alive, `Responding = True`, `MainWindowHandle` valid, **`MainWindowTitle` empty**, and **zero `msedgewebview2.exe` children** |
| 18:12 | Killed and restarted `buzz-desktop.exe` — the only way to recover; the WebView2 host is never re-created on its own |

Immediately after the restart:

| | before restart (t+6h) | after killing buzz-desktop | after restart (t+0) |
|---|---|---|---|
| commit charge | 65.1 GB / 73 GB | — | 28.5 GB |
| `node.exe`, system-wide | 131 | 7 | 5 under buzz-desktop |
| `claude.exe`, system-wide | 17 | 1 (my own terminal) | 1 under buzz-desktop |
| free physical RAM | 2.7 GB | — | 15.4 GB |

Killing `buzz-desktop.exe` released **36.5 GB of commit charge**, so the Buzz process tree owned more than half of the total commit at the time of failure. (I measured system-wide counts before the restart and only walked the process tree afterwards, so the pre-restart attribution to Buzz is inferred from what disappeared when the tree was killed — 124 `node.exe` and 16 `claude.exe` vanished with it.)

## Measurements

**Cost of a single ACP session.** Full descendant tree of one `claude.exe` worker (PID 53008), measured 16 minutes after a clean start:

```
claude.exe (1) + 33 descendants = 33 processes total, 2.06 GB commit

7 node.exe (MCP servers)
7 conhost.exe (Windows console host, one per cmd.exe)
6 cmd.exe (Windows shell wrapper per stdio MCP server)
4 python.exe
2 uvx.exe / 2 uv.exe
2 bun.exe
1 awslabs.aws-iac-mcp-server.exe
1 awslabs.aws-pricing-mcp-server.exe
```

Per-session cost is obviously a function of how many MCP servers the user configures — that part is on me, not on Buzz. What matters here is the **multiplier**: Buzz keeps one such tree alive per (agent, channel) pair, indefinitely.

**The multiplier is real, not hypothetical.** Ancestry chains of every `claude.exe` 18 minutes after the clean restart (the first line is my own unrelated terminal session):

```
18:06:00 | claude.exe(16396) <- powershell.exe <- (unrelated, my own shell)
18:14:04 | claude.exe(22188) <- node.exe(38932) <- cmd.exe <- buzz-acp.exe(42980) <- buzz-desktop.exe
18:14:14 | claude.exe(57116) <- node.exe(51636) <- cmd.exe <- buzz-acp.exe(59524) <- buzz-desktop.exe
18:14:28 | claude.exe(53008) <- node.exe(36484) <- cmd.exe <- buzz-acp.exe(27672) <- buzz-desktop.exe
18:18:55 | claude.exe(21928) <- node.exe(11648) <- cmd.exe <- buzz-acp.exe(50308) <- buzz-desktop.exe
18:19:05 | claude.exe(35768) <- node.exe(36484) <- cmd.exe <- buzz-acp.exe(27672) <- buzz-desktop.exe
18:30:15 | claude.exe(44044) <- node.exe(36484) <- cmd.exe <- buzz-acp.exe(27672) <- buzz-desktop.exe
18:30:47 | claude.exe(37028) <- node.exe(38932) <- cmd.exe <- buzz-acp.exe(42980) <- buzz-desktop.exe
```

`buzz-acp.exe(27672)` is configured with **`parallelism: 1`** and has a **single** adapter process, yet it has accumulated **three** live `claude.exe` children (18:14, 18:19, 18:30), none of which exited. `buzz-acp.exe(42980)` has two. New ones appear as new channels see their first message.

**The arithmetic reproduces the failure.** ~30 (agent, channel) pairs × 2.06 GB ≈ **62 GB**, against a 73 GB commit limit. The observed 65.1 GB at failure matches.

## Root-cause analysis

`SessionState` (`crates/buzz-acp/src/pool.rs`, quoted at tag **`v0.4.26`**; identical on `origin/main` `dd222a5`) maps **channel → ACP session**:

```rust
pub struct SessionState {
/// channel_id → session_id
pub sessions: HashMap,
...
}
```

- A session is created lazily on the first prompt for a channel (`pool.rs:1487`, `agent.state.sessions.insert(*cid, sid.clone())`).
- Entries are only ever removed by `invalidate_channel` (`pool.rs:123-127`) / `invalidate_all` — i.e. on **error, hard timeout, agent exit, or explicit rotation**. There is no idle eviction and no cap on the number of concurrent sessions.
- Proactive rotation exists but is **off by default**: `max_turns_per_session` defaults to `0` (`config.rs:372-374`), and rotation would only recycle a session, not bound the total.
- `idle_timeout_seconds` on the agent record is a *per-turn* silence guard (`acp.rs: session_prompt_with_idle_timeout`), not a session lifetime. Setting it does not help.
- Heartbeat sessions are not a factor here — `heartbeat_interval` defaults to `0` (`config.rs:297-298`).

On the adapter side, `claude-agent-acp` materialises each ACP session as its own `claude` CLI process with its own MCP server set. ACP as used here has `session/new`, `session/prompt`, `session/cancel` — but **no session close/end verb**, so even `invalidate_channel` only forgets the ID on the Buzz side; it gives the adapter no signal to tear the process down. Dropping a session from the map is therefore not just a no-op for memory, it is *strictly worse* — the old tree stays resident and the next prompt spawns another one.

So the resident set is:

```
live agent process trees ≈ Σ over agents ( channels that have seen ≥1 message )
```

monotonically non-decreasing for the lifetime of the worker, with no upper bound derived from `parallelism`.

## Why the failure mode is worse than "the machine ran out of memory"

When Windows hits the commit limit it terminates a victim process. The victim it picked was Buzz's own WebView2 host. Because Tauri's window and the Rust backend survive:

- the app looks alive (process running, `Responding = True`, window accepts drags)
- the screen is uniformly black
- **nothing is logged anywhere** — not `%APPDATA%\xyz.block.buzz.app\`, not the agent logs, not stdout
- WebView2 is **not** re-created, so the app never recovers on its own
- the only diagnosis path is comparing the OS process tree against the Windows system event log

A user hitting this has no way to tell it apart from a hang or a GPU problem. Detecting `ICoreWebView2` process failure (`ProcessFailed` event, which WebView2 exposes) and either recreating the WebView or showing a native "renderer terminated — restart?" dialog would turn a silent brick into a one-click recovery.

## Impact and scope

- **Affects:** any deployment where agents are members of several channels. It scales with `agents × channels`, which is precisely the direction a "fleet" deployment grows. It is worst with heavyweight adapters (`claude-agent-acp`, `codex-acp`) and MCP-rich agent configs.
- **Windows-specific amplification:** each stdio MCP server costs an extra `cmd.exe` + `conhost.exe` pair, so the per-session process count is roughly 2× what the same config costs on macOS/Linux. The 33-process figure above includes 13 such wrapper processes.
- **Not reproducible with a trivial config.** With one agent in one channel and no MCP servers this is invisible. It needs a multi-channel fleet to surface, which is why it may not have been seen in-house.
- **Relationship to #2631:** complementary, not duplicate. #2631 is about the *pool* (`parallelism` 24, eager init). This is about *sessions* (per channel, never released), and it reproduces with `parallelism: 1`. A fix for #2631 alone would not change any number in this report.

## Suggested directions (for maintainers to weigh — I have no strong opinion on which is right)

1. **Bound and evict sessions.** An LRU / idle-TTL over `SessionState.sessions` with a configurable max-live-sessions per worker. Since ACP has no close verb, eviction realistically means recycling the adapter process (as the hard-timeout path already does) — accepting the context loss for cold channels.
2. **Push a close upstream.** A `session/close` (or documented "drop the session's resources") in the ACP spec would let Buzz release adapter-side state without killing the worker. Larger, slower, but the clean fix.
3. **Make the cost visible.** Surface live session count and the owned process-tree footprint in the agent UI. Today there is no way for a user to see that 8 idle agents are holding 30 sessions and 60 GB.
4. **Fail loudly, not blackly.** Handle WebView2 `ProcessFailed` — recreate the WebView, or show a native dialog. This is worth doing regardless of the memory question, since any external termination produces the same silent black window.
5. **Document the resource model.** "Each channel an agent joins costs one live agent process tree" is load-bearing information for anyone sizing a self-hosted deployment, and it is not currently written down anywhere I could find.

## What I can provide

This is a live, reproducible environment (stock 0.4.26, self-hosted relay, 12-agent fleet, Windows). I am happy to:

- re-run any of the above measurements with a specific agent/channel configuration you want to see
- log session counts over time from an instrumented `buzz-acp` build if that would help
- test a candidate eviction patch under the same load

Say the word and I will run whatever measurement is most useful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.