anthropics / anthropics/claude-code

[BUG] Windows: peer-session liveness probe (powershell + Get-CimInstance Win32_Process, 1 s timeout) leaves unkillable orphan processes -> commit exhaustion -> crash 0xC0000409

Aperta
#93,274 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
area:core bug has repro platform:windows
Lingua principale
Python
Stelle
145k
Fork
23.1k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

**Versions affected (verified, probe code byte-identical):** 2.1.263, 2.1.266, 2.1.267
(VS Code / Antigravity extension bundle, `resources/native-binary/claude.exe`)
**OS:** Windows 11 Pro 10.0.26200, 32 GB RAM, HP Pavilion TP01 (Intel), Windows Defender + local AV
**Shell used by Claude Code for the probe:** Windows PowerShell 5.1 (`powershell.exe`)

---

## Summary

On Windows, Claude Code periodically checks whether peer sessions registered in
`%USERPROFILE%\.claude\sessions\..key` are still alive. The fast path
(`bun:ffi` → kernel32 `OpenProcess`/`GetProcessTimes`) is **unavailable in the bundled
binary on this machine**, so every check falls back to spawning:

```
powershell.exe -NoProfile -Command "(Get-CimInstance Win32_Process -Filter \"ProcessId=${e}\").CreationDate.Ticks"
```

with `{ timeout: 1000 }` on the Node side.

`Get-CimInstance Win32_Process -Filter "ProcessId=N"` is **not** an O(1) lookup. The
`cimwin32` provider enumerates **every** process on the box and filters afterwards,
at a measured **~1.45 ms per process**. On a developer machine with ~500–1,200
processes (Electron IDE, JVM language servers, browsers, MCP servers), the probe takes
**0.8–2.0 s**, so it exceeds the 1 s timeout **structurally**, on every round.

When Node kills `powershell.exe` while it is blocked inside the WMI/DCOM RPC, the
process **never finishes tearing down**: `HasExited = True`, `taskkill` reports "no
running instance", `Stop-Process -Force` returns success — yet each one keeps
**~43 MB of commit charge** and one thread in `ThreadState=5 (Wait) /
WaitReason=0 (Executive)`. Nothing except a reboot releases them.

Because **every session probes every peer key, with 3–4 retries on timeout**, the number
of probes per 15-minute round scales as roughly `sessions × peers × 3.5`. With 13–14
sessions open that is **~500–600 `powershell.exe` spawns per round**, which alone
produces a **+5 to +10 percentage-point commit spike** (each PowerShell ~43 MB). When
baseline commit is already high, that spike is what trips the process into the
fail-fast exit (`3221226505 / 0xC0000409`).

---

## Observed impact

| Metric | Value |
|---|---|
| Orphan probe processes accumulated (2.5 days, 17 sessions) | **574**, ~24 GB commit |
| Commit charge at crash | 96–97 % (of 102–129 GB limit) |
| Exit code | `3221226505` (`0xC0000409`, STATUS_STACK_BUFFER_OVERRUN / fail-fast) |
| WMI-Activity log (Event 5858), 4 h window | **402 probe failures** (`0x800706BA` RPC server unavailable = client killed mid-call), **540 cancellations** (`0x80041032`) |
| Probe processes per 15-min round observed with 2 sessions | 13 (same 3 target PIDs, 4–5× each = retries) |
| Zombie formation rate per killed probe | 5–15 % when PowerShell starts fast (kill lands inside RPC); ~1 % when start-up is slow (kill lands before RPC) |
| Commit spike per probe round at 13 sessions | +5 to +10 pp, 60–90 s duration |

Three crashes in two days; each requires a full reboot (not sleep/shutdown — Fast Startup
preserves the wedged kernel objects).

---

## Root cause chain (all steps measured)

1. **Fast path dead.** Binary contains both branches:
`[win32-proc-times] bun:ffi loaded, using procStartFt` and
`bun:ffi unavailable, falling back to spawn`. On this machine only the fallback ever runs
(0 probes would spawn PowerShell if FFI worked). Why FFI is unavailable in the VS Code
extension-bundled `claude.exe` on Windows was not determinable from outside.

2. **Fallback cost is linear in process count, provider-side.** Measured with a fresh
`powershell.exe` each time, return value verified:

| Processes on box | `Get-CimInstance Win32_Process -Filter ProcessId=N` |
|---:|---:|
| 572 | 790 ms |
| 723 | 1,003 ms (.NET) / 1,085 ms (native VBScript client — same) |
| 1,227 | 1,953 ms |

Independent of client (PowerShell, .NET, VBScript/WbemScripting), query shape
(`SELECT *`, `SELECT CreationDate`, `WHERE Handle=`, `WHERE ProcessId=`), DCOM
authentication level, or AV exclusions. It is the `cimwin32` instance construction plus
the two-hop marshalling (WmiPrvSE → Winmgmt → client).
A native `OpenProcess` + `GetProcessTimes` over the same 741 processes costs
**0.25–0.44 ms/proc**; WMI costs **2.3 ms/proc**.

3. **Start-up cost adds 276–900 ms.** `powershell.exe -NoProfile` cold start is ~276 ms with
a valid `System.Management.Automation` NGEN image, ~900 ms without one (Windows servicing
invalidated it on 2026-09-09; `ngen update` restored it).

4. **Budget:** `probe ≈ 276 + 1.45 × N (+ ~50–100 CIM init)`. For a 1,000 ms timeout the
machine needs **N ≲ 430 processes**. This machine's baseline with **zero** Claude
sessions is ~530 (IDE 76, JVM 28, Chrome 32, svchost ~100, …). A typical laptop with
~200 processes finishes in ~570 ms, which is why the bug is invisible in most
environments.

5. **Kill-mid-RPC → wedged process.** The killed `powershell.exe` has one thread stuck in an
ALPC/RPC executive wait; the address space (~43 MB commit) is never reclaimed.
`Stop-Process -Force` on 569 of them returned `terminated=569, failed=0`; **0** actually
went away.

6. **Self-acceleration.** Zombies ↑ → process count ↑ → enumeration slower → more timeouts
→ more zombies. This is why the machine is "fine for hours, then suddenly explodes".

7. **Stale keys amplify it.** After a reboot, a former Claude PID was reused by an unrelated
process (`codex.exe`); its `.key` file remained and was probed 5× per round.

8. **Not related to Remote Control.** `disableRemoteControl: true` /
`remoteControlAtStartup: false` had no effect (verified after reboot). The
`heartbeatIntervalMs` / `livenessTimer` constants in the binary belong to `CCRClient`
(the Remote Control transport); the PID probe is driven by the local cross-session
registry (`~/.claude/sessions/*.key`).

---

## How to reproduce

1. Windows 11, a machine with ≥ ~450 processes (e.g. open two Electron IDEs, a JVM project,
a browser with tabs). Check: `(Get-Process).Count`.
2. Open 3+ Claude Code sessions (tabs) in VS Code so that `%USERPROFILE%\.claude\sessions\`
contains 3+ `.key` files.
3. Wait 15–30 minutes. Then:

```powershell
$live = @{}; Get-Process | % { $live[$_.Id] = $true }
Get-CimInstance Win32_Process -Filter "Name='powershell.exe'" |
? { $_.CommandLine -like '*CreationDate.Ticks*' -and -not $live.ContainsKey($_.ParentProcessId) } |
Measure-Object | Select Count
```

Expected: a growing count of orphaned probes. `Stop-Process -Force` on them "succeeds"
but they remain in Task Manager (43 MB each, 0 CPU). Only a restart clears them.

4. Measure the probe latency the way Claude Code experiences it (fresh process):

```powershell
$sw=[Diagnostics.Stopwatch]::StartNew()
$p=Start-Process powershell -ArgumentList '-NoProfile','-Command','(Get-CimInstance Win32_Process -Filter "ProcessId=4").CreationDate.Ticks' -PassThru -NoNewWindow -RedirectStandardOutput "$env:TEMP\probe.txt"
$p.WaitForExit(); $sw.ElapsedMilliseconds; Get-Content "$env:TEMP\probe.txt"
```

On a ~700-process box this returns > 1,000 ms with a valid Ticks value — i.e. the probe
was *correct but too slow*, and would have been killed.

---

## Suggested fixes (any one of the first two removes the failure mode)

1. **Restore / harden the `bun:ffi` path** (kernel32 `OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION)` + `GetProcessTimes`). It is ~1,000× cheaper than WMI and cannot wedge. Please log *why* FFI is unavailable (dlopen error) so users can report it; the current fallback is silent.

2. **If a spawn fallback must remain, do not enumerate.** Options, all measured on the same box:
- `[wmi]"Win32_Process.Handle=N"` (key-path `GetObject`, no enumeration): **~201 ms** regardless of process count.
- `(Get-Process -Id N).StartTime.ToFileTime()` — no WMI at all; `Get-Process` over 723 processes takes 9 ms.
Either brings the probe to ~300–500 ms on any machine.

3. **Never hard-kill a WMI client mid-call.** If a timeout is kept, make it ≥ 5 s, or let the child exit on its own after the result is discarded, so the RPC completes and the process can tear down.

4. **Reduce probe volume.** Currently every session probes every peer with 3–4 retries per round (`≈ sessions × peers × 3.5`). Elect one prober per machine per round, or cache results in the registry with a TTL, and drop retries on timeout (a timeout is information, not a transient error here).

5. **Prune stale keys.** Before probing, verify the PID's image name is `claude.exe` (PID reuse after reboot); delete keys whose PID is dead or belongs to another image.

---

## Appendix — how the evidence was gathered

- Probe command and timeout: string search in the running `claude.exe`
(`Get-Process claude | Select Path` first — auto-update swaps the extension folder):
`CreationDate.Ticks` ×4, `timeout:1000` ×6, `bun:ffi unavailable, falling back to spawn` ×6,
identical across 2.1.266 and 2.1.267.
- Zombie state: `Get-Process -Id | Select HasExited`, thread state via
`Get-CimInstance Win32_Thread -Filter "ProcessHandle=''"` (ThreadState 5 / WaitReason 0),
commit via `Win32_Process.PageFileUsage` before/after `Stop-Process`.
- WMI failures: `Microsoft-Windows-WMI-Activity/Operational`, Event ID 5858, filtered on
`Win32_Process WHERE ProcessId=`; result codes `0x800706BA` / `0x80041032`.
- Round timing: zombie `CreationDate` values cluster at :14/:29/:44/:59 (15-minute cadence),
in pairs/groups matching the retry count.
- Commit charge: `Win32_OperatingSystem` TotalVirtualMemorySize − FreeVirtualMemory, sampled
every 60 s; the spike coincides with each probe round and scales with session count.

Happy to provide the raw logs (WMI-Activity export, 60-s commit time series, binary string
offsets) on request.

---

## Related issues

- #84675 — same spawn (`Get-CimInstance Win32_Process -Filter "ProcessId="` → `.CreationDate.Ticks`), reported there as a cosmetic visible-console-window problem. This report is the same code path's timeout / orphan / commit-exhaustion failure mode.
- #86551 — statusline `pwsh.exe` orphans under multi-session use on Windows; probably a different spawn site, but the same "killed child never finishes exiting" symptom.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start by locating the liveness-probe implementation corresponding to resources/native-binary/claude.exe and the reported bun:ffi fallback, then reproduce the timeout with the PowerShell Get-CimInstance command and the supplied orphan-process checks. Done should prevent probes from accumulating unkillable processes while preserving peer-session liveness checks; validate with the three-session reproduction and commit/process measurements.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
bun, powershell
Ambito
cli, operating-systems, performance
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Abbastanza chiara
Idoneità per principianti
42/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.