github / github/copilot-cli

Idle Copilot CLI enters `FileWatch` event storm, consumes two CPU cores, and writes a 33+ GB log

未关闭
#4,807 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

triage
主要语言
Shell
星标
11.2k
派生
1.9k
平均合并
14 小时 16 分钟
30 天内合并 PR
6

描述

Describe the bug

Summary

An idle Copilot CLI process launched by Agency entered a sustained FileWatch event storm. The process consumed approximately 221% CPU for more than 35 hours and continuously appended rejected file-watch events to its debug log.

The affected log was 33 GB when first investigated and later exceeded 37 GB. Measured log growth was approximately 216 KB/s, with total process writes around 292 KB/s.

Environment

  • Linux x86_64 under WSL2
  • Kernel: 5.15.167.4-microsoft-standard-WSL2
  • Copilot CLI: 1.0.82
  • Binary build ID: 2b4b8482cccada194dc991f2b088278437f71d7e
  • Process uptime when initially observed: approximately 35 hours
  • Process threads: 111
  • Agency session was open but idle

Actual behavior

  • Sustained process CPU: approximately 221%
  • Resident memory: approximately 1.4 GiB during initial profiling
  • Cumulative CPU time: approximately 77.6 hours after approximately 35 hours of process uptime
  • The last 10,000 log lines spanned approximately 7.1 seconds.
  • Of those lines, 9,982 were:
[DEBUG] [rust:copilot_runtime::protocol::jsonrpc::engine] No connection accepted a host event {"kind":"FileWatch"}

This is approximately 1,400 rejected FileWatch events per second.

No files under the session workspace had modification timestamps in the preceding two minutes, and the process had no active network sockets. This does not appear to be legitimate filesystem activity or active model work.

Other idle Copilot CLI 1.0.83 sessions on the same host consumed approximately 3-4% CPU each. The storm was isolated to the older 1.0.82 process.

Expected behavior

An idle session should not continuously consume multiple CPU cores or produce an unbounded stream of rejected file-watch events. Repeated equivalent events should be coalesced, rate-limited, or dropped without per-event debug logging.

Performance profile collection

Several complementary profiles were collected while the issue was continuously reproducible:

  1. Per-thread CPU and process-accounting snapshot
  2. Three repeated native stack snapshots using eu-stack
  3. Ten-second syscall profile using strace -f -c
  4. Thirty-second Samply attachment attempt at 99 Hz

Samply attached successfully after temporarily relaxing the Linux perf limits, but WSL returned zero samples. The Samply file is retained as evidence of that profiler limitation, but the actionable data is in the thread accounting, eu-stack, and strace captures.

Hot-thread accounting
TID State Instant CPU Cumulative CPU
832 R 54.5% unavailable
849 S 27.3% 101 minutes
853 R 18.2% 161 minutes
855 R 27.3% 123 minutes
858 S 18.2% 240 minutes

The work was distributed across the main thread and native runtime workers rather than being a transient single-thread spike.

Top native stacks

The main thread repeatedly appeared in this Node/V8 foreground-task path:

__memcpy_avx_unaligned_erms
...
v8::platform::DefaultJobState::Join()
v8::platform::DefaultJobHandle::Join()
...
node::PerIsolatePlatformData::FlushForegroundTasksInternal()
...
uv_run
node::SpinEventLoopInternal(node::Environment*)
node::NodeMainInstance::Run()
node::Start(int, char**)

Hot worker TIDs repeatedly showed this shape:

syscall
<frames inside prebuilds/linux-x64/runtime.node>
...
start_thread
__clone3

One repeated worker snapshot was actively writing:

write
<frames inside prebuilds/linux-x64/runtime.node>
...
start_thread
__clone3

The runtime.node binary is stripped, so internal Rust frames could not be symbolicated locally. The repeated main-thread V8 job join/task flush, hot native runtime workers, and continuous copilot_runtime::protocol::jsonrpc::engine logging strongly correlate the CPU consumption with runtime event dispatch and rejection rather than filesystem scanning alone.

Syscall profile

The ten-second strace -f -c capture across 111 threads recorded 88,712 calls:

Syscall Calls Errors
futex 52,893 6,863
write 10,646 0
read 9,648 1,420
epoll_pwait 6,225 0
epoll_wait 3,626 0
statx 2,718 0
openat 1,359 0
close 1,359 0

The high futex rate shows heavy worker synchronization, while more than 1,000 writes per second matches the observed per-event debug-log storm. The repeated openat/statx calls also indicate ongoing filesystem-related processing despite the workspace being idle.

Likely culprit

The strongest evidence points to a feedback loop or queued-event replay in the native Copilot runtime:

  1. FileWatch host events are generated at approximately 1,400 events/s.
  2. No connection accepts the events.
  3. Each rejection is logged individually.
  4. Native runtime.node workers remain hot and heavily synchronized.
  5. The Node main thread repeatedly joins V8 jobs and flushes foreground tasks.
  6. No corresponding workspace changes or network activity explain the event rate.

Potential areas to investigate:

  • File-watch event production and deduplication
  • Host-event dispatch when no connection accepts an event
  • Requeue/retry behavior for rejected host events
  • Per-event debug logging without rate limiting
  • Whether the log write or another watched path can feed new events back into the watcher

Reproduction information

The issue was observed in an Agency-launched Copilot CLI 1.0.82 session left open and idle for approximately 35 hours. A deterministic trigger is not yet known. Once present, the behavior persisted indefinitely and remained reproducible throughout profiling.

Affected version

No response

Steps to reproduce the behavior

No response

Expected behavior

No response

Additional context

No response

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

从 copilot_runtime::protocol::jsonrpc::engine 的 logging 路径,以及报告中描述的 file-watch 事件生成、分发和重试区域开始。优先使用 eu-stack 和 strace 的发现;Samply profile 没有可用的 samples。完成标准是:空闲 session 不再产生持续的被拒绝 FileWatch 事件风暴、多核 CPU 使用率,或无界的逐事件 logging。

由索引模型根据 Issue 内容生成。

评估

技术栈
node.js, rust
领域
cli, performance
Issue 类型
缺陷
难度
5/5
预计耗时
一周以上
活跃度
活跃
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。