anthropics / anthropics/claude-code

[BUG] Windows: peer-session liveness probe (powershell + Get-CimInstance Win32_Process, 1 s timeout) leaves unkillable orphan processes -> commit exhaustion -> crash 0xC0000409

Abierto
#93,274 0 comentarios 0 reacciones 0 asignados Ver en GitHub
area:core bug has repro platform:windows
Lenguaje dominante
Python
Estrellas
145k
Forks
23.1k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

**Versions affected (verified, probe code byte-identical):** 2.1.263, 2.1.266, 2.1.267
(VS Code / Antigravity extension bundle, `resources/native-binary/claude.exe`)
**OS:** Windows 11 Pro 10.0.26200, 32 GB RAM, HP Pavilion TP01 (Intel), Windows Defender + local AV
**Shell used by Claude Code for the probe:** Windows PowerShell 5.1 (`powershell.exe`)

---

## Summary

On Windows, Claude Code periodically checks whether peer sessions registered in
`%USERPROFILE%\.claude\sessions\..key` are still alive. The fast path
(`bun:ffi` → kernel32 `OpenProcess`/`GetProcessTimes`) is **unavailable in the bundled
binary on this machine**, so every check falls back to spawning:

```
powershell.exe -NoProfile -Command "(Get-CimInstance Win32_Process -Filter \"ProcessId=${e}\").CreationDate.Ticks"
```

with `{ timeout: 1000 }` on the Node side.

`Get-CimInstance Win32_Process -Filter "ProcessId=N"` is **not** an O(1) lookup. The
`cimwin32` provider enumerates **every** process on the box and filters afterwards,
at a measured **~1.45 ms per process**. On a developer machine with ~500–1,200
processes (Electron IDE, JVM language servers, browsers, MCP servers), the probe takes
**0.8–2.0 s**, so it exceeds the 1 s timeout **structurally**, on every round.

When Node kills `powershell.exe` while it is blocked inside the WMI/DCOM RPC, the
process **never finishes tearing down**: `HasExited = True`, `taskkill` reports "no
running instance", `Stop-Process -Force` returns success — yet each one keeps
**~43 MB of commit charge** and one thread in `ThreadState=5 (Wait) /
WaitReason=0 (Executive)`. Nothing except a reboot releases them.

Because **every session probes every peer key, with 3–4 retries on timeout**, the number
of probes per 15-minute round scales as roughly `sessions × peers × 3.5`. With 13–14
sessions open that is **~500–600 `powershell.exe` spawns per round**, which alone
produces a **+5 to +10 percentage-point commit spike** (each PowerShell ~43 MB). When
baseline commit is already high, that spike is what trips the process into the
fail-fast exit (`3221226505 / 0xC0000409`).

---

## Observed impact

| Metric | Value |
|---|---|
| Orphan probe processes accumulated (2.5 days, 17 sessions) | **574**, ~24 GB commit |
| Commit charge at crash | 96–97 % (of 102–129 GB limit) |
| Exit code | `3221226505` (`0xC0000409`, STATUS_STACK_BUFFER_OVERRUN / fail-fast) |
| WMI-Activity log (Event 5858), 4 h window | **402 probe failures** (`0x800706BA` RPC server unavailable = client killed mid-call), **540 cancellations** (`0x80041032`) |
| Probe processes per 15-min round observed with 2 sessions | 13 (same 3 target PIDs, 4–5× each = retries) |
| Zombie formation rate per killed probe | 5–15 % when PowerShell starts fast (kill lands inside RPC); ~1 % when start-up is slow (kill lands before RPC) |
| Commit spike per probe round at 13 sessions | +5 to +10 pp, 60–90 s duration |

Three crashes in two days; each requires a full reboot (not sleep/shutdown — Fast Startup
preserves the wedged kernel objects).

---

## Root cause chain (all steps measured)

1. **Fast path dead.** Binary contains both branches:
`[win32-proc-times] bun:ffi loaded, using procStartFt` and
`bun:ffi unavailable, falling back to spawn`. On this machine only the fallback ever runs
(0 probes would spawn PowerShell if FFI worked). Why FFI is unavailable in the VS Code
extension-bundled `claude.exe` on Windows was not determinable from outside.

2. **Fallback cost is linear in process count, provider-side.** Measured with a fresh
`powershell.exe` each time, return value verified:

| Processes on box | `Get-CimInstance Win32_Process -Filter ProcessId=N` |
|---:|---:|
| 572 | 790 ms |
| 723 | 1,003 ms (.NET) / 1,085 ms (native VBScript client — same) |
| 1,227 | 1,953 ms |

Independent of client (PowerShell, .NET, VBScript/WbemScripting), query shape
(`SELECT *`, `SELECT CreationDate`, `WHERE Handle=`, `WHERE ProcessId=`), DCOM
authentication level, or AV exclusions. It is the `cimwin32` instance construction plus
the two-hop marshalling (WmiPrvSE → Winmgmt → client).
A native `OpenProcess` + `GetProcessTimes` over the same 741 processes costs
**0.25–0.44 ms/proc**; WMI costs **2.3 ms/proc**.

3. **Start-up cost adds 276–900 ms.** `powershell.exe -NoProfile` cold start is ~276 ms with
a valid `System.Management.Automation` NGEN image, ~900 ms without one (Windows servicing
invalidated it on 2026-09-09; `ngen update` restored it).

4. **Budget:** `probe ≈ 276 + 1.45 × N (+ ~50–100 CIM init)`. For a 1,000 ms timeout the
machine needs **N ≲ 430 processes**. This machine's baseline with **zero** Claude
sessions is ~530 (IDE 76, JVM 28, Chrome 32, svchost ~100, …). A typical laptop with
~200 processes finishes in ~570 ms, which is why the bug is invisible in most
environments.

5. **Kill-mid-RPC → wedged process.** The killed `powershell.exe` has one thread stuck in an
ALPC/RPC executive wait; the address space (~43 MB commit) is never reclaimed.
`Stop-Process -Force` on 569 of them returned `terminated=569, failed=0`; **0** actually
went away.

6. **Self-acceleration.** Zombies ↑ → process count ↑ → enumeration slower → more timeouts
→ more zombies. This is why the machine is "fine for hours, then suddenly explodes".

7. **Stale keys amplify it.** After a reboot, a former Claude PID was reused by an unrelated
process (`codex.exe`); its `.key` file remained and was probed 5× per round.

8. **Not related to Remote Control.** `disableRemoteControl: true` /
`remoteControlAtStartup: false` had no effect (verified after reboot). The
`heartbeatIntervalMs` / `livenessTimer` constants in the binary belong to `CCRClient`
(the Remote Control transport); the PID probe is driven by the local cross-session
registry (`~/.claude/sessions/*.key`).

---

## How to reproduce

1. Windows 11, a machine with ≥ ~450 processes (e.g. open two Electron IDEs, a JVM project,
a browser with tabs). Check: `(Get-Process).Count`.
2. Open 3+ Claude Code sessions (tabs) in VS Code so that `%USERPROFILE%\.claude\sessions\`
contains 3+ `.key` files.
3. Wait 15–30 minutes. Then:

```powershell
$live = @{}; Get-Process | % { $live[$_.Id] = $true }
Get-CimInstance Win32_Process -Filter "Name='powershell.exe'" |
? { $_.CommandLine -like '*CreationDate.Ticks*' -and -not $live.ContainsKey($_.ParentProcessId) } |
Measure-Object | Select Count
```

Expected: a growing count of orphaned probes. `Stop-Process -Force` on them "succeeds"
but they remain in Task Manager (43 MB each, 0 CPU). Only a restart clears them.

4. Measure the probe latency the way Claude Code experiences it (fresh process):

```powershell
$sw=[Diagnostics.Stopwatch]::StartNew()
$p=Start-Process powershell -ArgumentList '-NoProfile','-Command','(Get-CimInstance Win32_Process -Filter "ProcessId=4").CreationDate.Ticks' -PassThru -NoNewWindow -RedirectStandardOutput "$env:TEMP\probe.txt"
$p.WaitForExit(); $sw.ElapsedMilliseconds; Get-Content "$env:TEMP\probe.txt"
```

On a ~700-process box this returns > 1,000 ms with a valid Ticks value — i.e. the probe
was *correct but too slow*, and would have been killed.

---

## Suggested fixes (any one of the first two removes the failure mode)

1. **Restore / harden the `bun:ffi` path** (kernel32 `OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION)` + `GetProcessTimes`). It is ~1,000× cheaper than WMI and cannot wedge. Please log *why* FFI is unavailable (dlopen error) so users can report it; the current fallback is silent.

2. **If a spawn fallback must remain, do not enumerate.** Options, all measured on the same box:
- `[wmi]"Win32_Process.Handle=N"` (key-path `GetObject`, no enumeration): **~201 ms** regardless of process count.
- `(Get-Process -Id N).StartTime.ToFileTime()` — no WMI at all; `Get-Process` over 723 processes takes 9 ms.
Either brings the probe to ~300–500 ms on any machine.

3. **Never hard-kill a WMI client mid-call.** If a timeout is kept, make it ≥ 5 s, or let the child exit on its own after the result is discarded, so the RPC completes and the process can tear down.

4. **Reduce probe volume.** Currently every session probes every peer with 3–4 retries per round (`≈ sessions × peers × 3.5`). Elect one prober per machine per round, or cache results in the registry with a TTL, and drop retries on timeout (a timeout is information, not a transient error here).

5. **Prune stale keys.** Before probing, verify the PID's image name is `claude.exe` (PID reuse after reboot); delete keys whose PID is dead or belongs to another image.

---

## Appendix — how the evidence was gathered

- Probe command and timeout: string search in the running `claude.exe`
(`Get-Process claude | Select Path` first — auto-update swaps the extension folder):
`CreationDate.Ticks` ×4, `timeout:1000` ×6, `bun:ffi unavailable, falling back to spawn` ×6,
identical across 2.1.266 and 2.1.267.
- Zombie state: `Get-Process -Id | Select HasExited`, thread state via
`Get-CimInstance Win32_Thread -Filter "ProcessHandle=''"` (ThreadState 5 / WaitReason 0),
commit via `Win32_Process.PageFileUsage` before/after `Stop-Process`.
- WMI failures: `Microsoft-Windows-WMI-Activity/Operational`, Event ID 5858, filtered on
`Win32_Process WHERE ProcessId=`; result codes `0x800706BA` / `0x80041032`.
- Round timing: zombie `CreationDate` values cluster at :14/:29/:44/:59 (15-minute cadence),
in pairs/groups matching the retry count.
- Commit charge: `Win32_OperatingSystem` TotalVirtualMemorySize − FreeVirtualMemory, sampled
every 60 s; the spike coincides with each probe round and scales with session count.

Happy to provide the raw logs (WMI-Activity export, 60-s commit time series, binary string
offsets) on request.

---

## Related issues

- #84675 — same spawn (`Get-CimInstance Win32_Process -Filter "ProcessId="` → `.CreationDate.Ticks`), reported there as a cosmetic visible-console-window problem. This report is the same code path's timeout / orphan / commit-exhaustion failure mode.
- #86551 — statusline `pwsh.exe` orphans under multi-session use on Windows; probably a different spawn site, but the same "killed child never finishes exiting" symptom.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Start by locating the liveness-probe implementation corresponding to resources/native-binary/claude.exe and the reported bun:ffi fallback, then reproduce the timeout with the PowerShell Get-CimInstance command and the supplied orphan-process checks. Done should prevent probes from accumulating unkillable processes while preserving peer-session liveness checks; validate with the three-session reproduction and commit/process measurements.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
bun, powershell
Área
cli, operating-systems, performance
Tipo de issue
Error
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Activo
Claridad
Bastante claro
Aptitud para principiantes
42/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.