github / github/copilot-cli

v1.0.80: event-storage exhaustion retry storm drives long-running session into GC/compaction loop and Node OOM

オープン
#4,639 コメント 2 件 リアクション 2 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

triage
主要言語
Shell
スター
11.2k
フォーク
1.9k
平均マージ
14時間 16分
マージ済み PR(30日)
6

説明

Describe the bug

A long-running active Copilot CLI session eventually reached remote event-storage exhaustion. After that, the exporter kept attempting 500-event flushes, while the process repeatedly reported memory pressure, forced GC/emergency compaction, and thousands of bridge acknowledgement timeouts. The CLI ultimately terminated with a Node out-of-memory error shown in the terminal.

This appears to connect two existing failure modes that are currently reported separately:

  • #4467: session_event_storage_exhausted
  • #4506: memory-pressure GC/compaction loop ending in OOM
  • Also related to #4251 for unbounded memory growth in large sessions

The new observation is that remote storage exhaustion does not merely make session status unreliable: the repeated exporter failures/backlog correlate with sustained local memory pressure and eventual process death.

Environment
  • Copilot CLI at failure: 1.0.80
  • Embedded Node.js: v24.18.1
  • OS: Linux x86_64, kernel 6.17.0-1022-azure
  • Host memory: 125 GiB RAM, no swap
  • Session created: 2026-08-10
  • Process/resume started: 2026-08-18
  • Failure: 2026-08-27 (about 8.8 days of this process, about 16.9 days of session lifetime)
  • Workload: long-running interactive/autonomous software-engineering session with many tool calls and subagents
Session size at failure
  • events.jsonl: 934,639,321 bytes (~892 MiB)
  • Event records: 282,912
  • Checkpoints: 32
Log evidence

Across the process log:

Event Count
session_event_storage_exhausted 18,909
Memory pressure detected - requesting garbage collection 21,352
Memory pressure persists after GC - triggering emergency compaction 473
timed out waiting for bridge event ack 3,001

Representative final minutes:

[WARNING] Failed to submit events to Mission Control session …: 409
{"code":"session_event_storage_exhausted","message":"session event storage exhausted"}
[WARNING] remote session batch flush failed {"event_count":500,...}
[WARNING] remote session exporter circuit opened after batch flush failure
[WARNING] Memory pressure detected - requesting garbage collection
[WARNING] Memory pressure persists after GC - triggering emergency compaction
[WARNING] timed out waiting for bridge event ack {"timeout_ms":30000}
[INFO] Compacted 17 messages, saved ~9267 tokens
[WARNING] Memory pressure detected - requesting garbage collection
...
[INFO] Compacted 38 messages, saved ~23339 tokens
[WARNING] Memory pressure detected - requesting garbage collection

The application log ends in repeated memory-pressure warnings. The fatal Node OOM text was printed by the terminating runtime to the terminal and was not persisted into the Copilot process log. No kernel OOM-killer event was recorded, so this was a Node/V8 process failure rather than the Linux kernel killing the process.

The host currently has ample free memory after restart; I did not capture /proc/<pid>/status immediately before the crash, so I cannot claim whether the final allocation failure was at the V8 heap limit or in native/external memory.

Steps to reproduce
  1. Keep one Copilot CLI session active/resumable for more than a week with frequent tool calls and subagent events.
  2. Allow events.jsonl to grow toward 1 GB and remote Mission Control event storage to fill.
  3. Observe repeated 409 session_event_storage_exhausted responses for 500-event batches.
  4. Continue using the session.
  5. Observe repeated memory-pressure GC, emergency compaction, bridge ack timeouts, and eventually a Node OOM termination.
Expected behavior
  • Rotate/continue/compact remote event storage before its limit is reached.
  • Once the server permanently rejects a stream as exhausted, stop retaining/retrying the rejected backlog in a way that grows local memory.
  • Keep local session operation and checkpointing healthy when remote export is unavailable.
  • Apply bounded backoff and expose a clear degraded-export state.
  • Do not repeatedly perform lossy conversation compaction unless it measurably relieves the memory pressure.
  • Persist the fatal Node diagnostic report path or equivalent crash telemetry in the process log.
Actual behavior

A permanent remote quota condition produced 18,909 failed submissions, concurrent memory-pressure/compaction loops, degraded bridge responsiveness, and eventual Node OOM. Recovery required starting a new session and manually reading the old checkpoint and session directory.

コントリビューションガイド

コントリビューションガイドを開く

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

調査の方向性

リモートセッションエクスポーターの500イベントflush失敗、session_event_storage_exhaustedレスポンス、および関連するprocess-logのメモリプレッシャーメッセージから調査を始めてください。拒否されたバッチがどのように保持され、再試行されるかを追跡し、その後、永続的なエクスポート失敗によってローカルセッションの動作が健全なまま維持され、上限のある再試行動作が使われ、OOMに至ることなく明確な縮退状態が記録されることを確認してください。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
node.js
領域
backend, cli, performance
issue の種類
バグ
難易度
5/5
見積もり時間
1週間以上
活発さ
活発
明瞭さ
おおむね明確
初心者へのやさしさ
28/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。