v1.0.80: event-storage exhaustion retry storm drives long-running session into GC/compaction loop and Node OOM
まだ誰も着手していません。
- 主要言語
- Shell
- スター
- 11.2k
- フォーク
- 1.9k
- 平均マージ
- 14時間 16分
- マージ済み PR(30日)
- 6
説明
Describe the bug
A long-running active Copilot CLI session eventually reached remote event-storage exhaustion. After that, the exporter kept attempting 500-event flushes, while the process repeatedly reported memory pressure, forced GC/emergency compaction, and thousands of bridge acknowledgement timeouts. The CLI ultimately terminated with a Node out-of-memory error shown in the terminal.
This appears to connect two existing failure modes that are currently reported separately:
- #4467:
session_event_storage_exhausted - #4506: memory-pressure GC/compaction loop ending in OOM
- Also related to #4251 for unbounded memory growth in large sessions
The new observation is that remote storage exhaustion does not merely make session status unreliable: the repeated exporter failures/backlog correlate with sustained local memory pressure and eventual process death.
Environment
- Copilot CLI at failure: 1.0.80
- Embedded Node.js: v24.18.1
- OS: Linux x86_64, kernel 6.17.0-1022-azure
- Host memory: 125 GiB RAM, no swap
- Session created: 2026-08-10
- Process/resume started: 2026-08-18
- Failure: 2026-08-27 (about 8.8 days of this process, about 16.9 days of session lifetime)
- Workload: long-running interactive/autonomous software-engineering session with many tool calls and subagents
Session size at failure
events.jsonl: 934,639,321 bytes (~892 MiB)- Event records: 282,912
- Checkpoints: 32
Log evidence
Across the process log:
| Event | Count |
|---|---|
session_event_storage_exhausted |
18,909 |
Memory pressure detected - requesting garbage collection |
21,352 |
Memory pressure persists after GC - triggering emergency compaction |
473 |
timed out waiting for bridge event ack |
3,001 |
Representative final minutes:
[WARNING] Failed to submit events to Mission Control session …: 409
{"code":"session_event_storage_exhausted","message":"session event storage exhausted"}
[WARNING] remote session batch flush failed {"event_count":500,...}
[WARNING] remote session exporter circuit opened after batch flush failure
[WARNING] Memory pressure detected - requesting garbage collection
[WARNING] Memory pressure persists after GC - triggering emergency compaction
[WARNING] timed out waiting for bridge event ack {"timeout_ms":30000}
[INFO] Compacted 17 messages, saved ~9267 tokens
[WARNING] Memory pressure detected - requesting garbage collection
...
[INFO] Compacted 38 messages, saved ~23339 tokens
[WARNING] Memory pressure detected - requesting garbage collection
The application log ends in repeated memory-pressure warnings. The fatal Node OOM text was printed by the terminating runtime to the terminal and was not persisted into the Copilot process log. No kernel OOM-killer event was recorded, so this was a Node/V8 process failure rather than the Linux kernel killing the process.
The host currently has ample free memory after restart; I did not capture /proc/<pid>/status immediately before the crash, so I cannot claim whether the final allocation failure was at the V8 heap limit or in native/external memory.
Steps to reproduce
- Keep one Copilot CLI session active/resumable for more than a week with frequent tool calls and subagent events.
- Allow
events.jsonlto grow toward 1 GB and remote Mission Control event storage to fill. - Observe repeated 409
session_event_storage_exhaustedresponses for 500-event batches. - Continue using the session.
- Observe repeated memory-pressure GC, emergency compaction, bridge ack timeouts, and eventually a Node OOM termination.
Expected behavior
- Rotate/continue/compact remote event storage before its limit is reached.
- Once the server permanently rejects a stream as exhausted, stop retaining/retrying the rejected backlog in a way that grows local memory.
- Keep local session operation and checkpointing healthy when remote export is unavailable.
- Apply bounded backoff and expose a clear degraded-export state.
- Do not repeatedly perform lossy conversation compaction unless it measurably relieves the memory pressure.
- Persist the fatal Node diagnostic report path or equivalent crash telemetry in the process log.
Actual behavior
A permanent remote quota condition produced 18,909 failed submissions, concurrent memory-pressure/compaction loops, degraded bridge responsiveness, and eventual Node OOM. Recovery required starting a new session and manually reading the old checkpoint and session directory.
コントリビューションガイド
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
調査の方向性
リモートセッションエクスポーターの500イベントflush失敗、session_event_storage_exhaustedレスポンス、および関連するprocess-logのメモリプレッシャーメッセージから調査を始めてください。拒否されたバッチがどのように保持され、再試行されるかを追跡し、その後、永続的なエクスポート失敗によってローカルセッションの動作が健全なまま維持され、上限のある再試行動作が使われ、OOMに至ることなく明確な縮退状態が記録されることを確認してください。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- node.js
- 領域
- backend, cli, performance
- issue の種類
- バグ
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 活発さ
- 活発
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 28/100