github / github/copilot-cli

Idle Copilot CLI enters `FileWatch` event storm, consumes two CPU cores, and writes a 33+ GB log

Aperta
#4,807 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

triage
Lingua principale
Shell
Stelle
11.2k
Fork
1.9k
Merge medio
14h 16m
PR unite (30g)
6

Descrizione

Describe the bug

Summary

An idle Copilot CLI process launched by Agency entered a sustained FileWatch event storm. The process consumed approximately 221% CPU for more than 35 hours and continuously appended rejected file-watch events to its debug log.

The affected log was 33 GB when first investigated and later exceeded 37 GB. Measured log growth was approximately 216 KB/s, with total process writes around 292 KB/s.

Environment

  • Linux x86_64 under WSL2
  • Kernel: 5.15.167.4-microsoft-standard-WSL2
  • Copilot CLI: 1.0.82
  • Binary build ID: 2b4b8482cccada194dc991f2b088278437f71d7e
  • Process uptime when initially observed: approximately 35 hours
  • Process threads: 111
  • Agency session was open but idle

Actual behavior

  • Sustained process CPU: approximately 221%
  • Resident memory: approximately 1.4 GiB during initial profiling
  • Cumulative CPU time: approximately 77.6 hours after approximately 35 hours of process uptime
  • The last 10,000 log lines spanned approximately 7.1 seconds.
  • Of those lines, 9,982 were:
[DEBUG] [rust:copilot_runtime::protocol::jsonrpc::engine] No connection accepted a host event {"kind":"FileWatch"}

This is approximately 1,400 rejected FileWatch events per second.

No files under the session workspace had modification timestamps in the preceding two minutes, and the process had no active network sockets. This does not appear to be legitimate filesystem activity or active model work.

Other idle Copilot CLI 1.0.83 sessions on the same host consumed approximately 3-4% CPU each. The storm was isolated to the older 1.0.82 process.

Expected behavior

An idle session should not continuously consume multiple CPU cores or produce an unbounded stream of rejected file-watch events. Repeated equivalent events should be coalesced, rate-limited, or dropped without per-event debug logging.

Performance profile collection

Several complementary profiles were collected while the issue was continuously reproducible:

  1. Per-thread CPU and process-accounting snapshot
  2. Three repeated native stack snapshots using eu-stack
  3. Ten-second syscall profile using strace -f -c
  4. Thirty-second Samply attachment attempt at 99 Hz

Samply attached successfully after temporarily relaxing the Linux perf limits, but WSL returned zero samples. The Samply file is retained as evidence of that profiler limitation, but the actionable data is in the thread accounting, eu-stack, and strace captures.

Hot-thread accounting
TID State Instant CPU Cumulative CPU
832 R 54.5% unavailable
849 S 27.3% 101 minutes
853 R 18.2% 161 minutes
855 R 27.3% 123 minutes
858 S 18.2% 240 minutes

The work was distributed across the main thread and native runtime workers rather than being a transient single-thread spike.

Top native stacks

The main thread repeatedly appeared in this Node/V8 foreground-task path:

__memcpy_avx_unaligned_erms
...
v8::platform::DefaultJobState::Join()
v8::platform::DefaultJobHandle::Join()
...
node::PerIsolatePlatformData::FlushForegroundTasksInternal()
...
uv_run
node::SpinEventLoopInternal(node::Environment*)
node::NodeMainInstance::Run()
node::Start(int, char**)

Hot worker TIDs repeatedly showed this shape:

syscall
<frames inside prebuilds/linux-x64/runtime.node>
...
start_thread
__clone3

One repeated worker snapshot was actively writing:

write
<frames inside prebuilds/linux-x64/runtime.node>
...
start_thread
__clone3

The runtime.node binary is stripped, so internal Rust frames could not be symbolicated locally. The repeated main-thread V8 job join/task flush, hot native runtime workers, and continuous copilot_runtime::protocol::jsonrpc::engine logging strongly correlate the CPU consumption with runtime event dispatch and rejection rather than filesystem scanning alone.

Syscall profile

The ten-second strace -f -c capture across 111 threads recorded 88,712 calls:

Syscall Calls Errors
futex 52,893 6,863
write 10,646 0
read 9,648 1,420
epoll_pwait 6,225 0
epoll_wait 3,626 0
statx 2,718 0
openat 1,359 0
close 1,359 0

The high futex rate shows heavy worker synchronization, while more than 1,000 writes per second matches the observed per-event debug-log storm. The repeated openat/statx calls also indicate ongoing filesystem-related processing despite the workspace being idle.

Likely culprit

The strongest evidence points to a feedback loop or queued-event replay in the native Copilot runtime:

  1. FileWatch host events are generated at approximately 1,400 events/s.
  2. No connection accepts the events.
  3. Each rejection is logged individually.
  4. Native runtime.node workers remain hot and heavily synchronized.
  5. The Node main thread repeatedly joins V8 jobs and flushes foreground tasks.
  6. No corresponding workspace changes or network activity explain the event rate.

Potential areas to investigate:

  • File-watch event production and deduplication
  • Host-event dispatch when no connection accepts an event
  • Requeue/retry behavior for rejected host events
  • Per-event debug logging without rate limiting
  • Whether the log write or another watched path can feed new events back into the watcher

Reproduction information

The issue was observed in an Agency-launched Copilot CLI 1.0.82 session left open and idle for approximately 35 hours. A deterministic trigger is not yet known. Once present, the behavior persisted indefinitely and remained reproducible throughout profiling.

Affected version

No response

Steps to reproduce the behavior

No response

Expected behavior

No response

Additional context

No response

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia dal percorso di logging di copilot_runtime::protocol::jsonrpc::engine e dalle aree di produzione, dispatch e retry degli eventi file-watch descritte nel report. Usa prima i risultati di eu-stack e strace; il profilo Samply non contiene samples utilizzabili. Il lavoro è completato quando una sessione inattiva non produce più una raffica persistente di eventi FileWatch rifiutati, un utilizzo della CPU su più core o un logging illimitato per evento.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
node.js, rust
Ambito
cli, performance
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.