github / github/copilot-cli

Headless server leaks kqueue file descriptors — bash tool stops working after prolonged use

Đang mở
#2,389 2 bình luận 0 reaction 1 người được giao Được giao cho @MackinnonBuck Xem trên GitHub
area:non-interactive area:tools
Ngôn ngữ chính
Shell
Star
11.2k
Fork
1.9k
Merge trung bình
14 giờ 16 phút
Pull request đã merge (30 ngày)
6

Mô tả

## Summary

The copilot headless server (`copilot --headless`) accumulates kqueue file descriptors over its lifetime. Each session creates `fs.watch()` file watchers that are never released when sessions become idle or are closed. After enough sessions (especially with multi-agent/fleet workflows), PTY allocation for the `bash` tool fails with the generic error `"Failed to start bash process"`.

## Environment

- **CLI version:** 1.0.10
- **OS:** macOS 15.3 (Darwin 25.3.0, arm64)
- **Node:** v24.11.1
- **Client type:** cli-server (PolyPilot app connecting via TCP)

## Reproduction

1. Start `copilot --headless --port 4321`
2. Create many sessions over time (fleet/multi-agent workflows accelerate this — we had 93 unique sessions over the server's lifetime)
3. After enough sessions, all `bash` tool calls fail with `"Failed to start bash process"`
4. Existing PTY handles start throwing `EIO` (I/O error) on write

Restarting the headless server immediately fixes the issue.

## Evidence

### kqueue FD leak comparison

| Metric | Old Server (leaked) | Fresh Server (just restarted) |
|---|---|---|
| **Total FDs** | **4,321** | 69 |
| **KQUEUE FDs** | **9,779** | 27 |
| **Node processes** | 300 | 21 |
| **User processes** | 922 | 362 |

That's ~105 kqueue file descriptors leaked per session (9,779 kqueues / 93 sessions).

A fresh server starts with ~27 KQUEUE FDs. After 93 sessions (including multi-agent sub-agents), this grew to 9,779 — a **362x increase**.

### Timeline from the process log (`process-*.log`)

- **First PTY EIO error:** `2026-03-29T14:38:07.722Z`
```
[ERROR] Unhandled pty write error [Error: EIO: i/o error, write] {
errno: -5,
code: 'EIO',
syscall: 'write'
}
```
- **First bash failure:** `2026-03-29T14:38:07.925Z` (simultaneous with EIO)
```json
{
"tool_name": "bash",
"result_type": "FAILURE",
"error": ""
}
```
- **Last bash failure:** `2026-03-29T14:46:32.763Z` (~8 minutes of total bash unavailability)
- **Total PTY EIO errors:** 149
- **Total bash spawn failures:** 641
- **Multiple sessions affected** — the failure is server-wide, not session-specific

### lsof output on the leaked server (before restart)

```
$ lsof -p 26081 | awk '{print $5}' | sort | uniq -c | sort -rn
9779 KQUEUE
436 unix
120 CHR
27 REG
8 DIR
6 PIPE
5 IPv4
2 IPv6
1 systm
```

### lsof output on fresh server (after restart)

```
$ lsof -p 33499 | awk '{print $5}' | sort | uniq -c | sort -rn
27 KQUEUE
17 REG
6 PIPE
4 unix
4 IPv4
4 DIR
2 IPv6
2 CHR
1 systm
```

## Root Cause Analysis

Node.js uses kqueue on macOS for `fs.watch()`. The headless server likely creates file watchers per session (working directory monitoring, `.copilot/` config watches, session state directory watches, etc.). When sessions go idle or are explicitly closed, **these watchers are never cleaned up**.

With multi-agent workflows that spawn many short-lived sub-agent sessions (fleet mode, parallel task execution), the leak accelerates rapidly. In our case, 93 sessions accumulated ~9,779 kqueue FDs.

When the kqueue count gets high enough, `pty.spawn()` (used by the bash tool to create pseudo-terminal sessions) fails — likely due to macOS kernel resource pressure on PTY allocation or the child process inheriting too many FDs.

## Impact

- **All bash tool calls fail server-wide** — not just the session that triggered the limit
- **Existing PTY sessions get EIO errors** — even previously-working bash sessions break
- **Only fix is server restart** — the FDs are never reclaimed without killing the process
- **Multi-agent/fleet workflows hit this faster** due to many short-lived sessions
- **8+ minutes of total bash unavailability** in our observed incident

## Suggested Fix

1. **Close file watchers when sessions are disposed/idle.** Each `fs.watch()` handle should be tracked per-session and `.close()`d when the session ends.
2. **Set `CLOEXEC` on kqueue FDs** if not already — prevents child processes (bash) from inheriting unnecessary FDs.
3. **Add a resource limit guard** — if the server detects its FD count exceeding a threshold, log a warning and/or proactively clean up stale watchers.

## Workaround

Restart the headless server periodically, or when bash failures are detected. PolyPilot users can use Settings → Save & Reconnect to trigger a server restart.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.