Keep a persistent append handle for `events.jsonl` instead of open/append/close per event (Windows Defender rescan / CPU cost)
- Dominant language
- Shell
- Stars
- 11.2k
- Forks
- 1.9k
- Avg merge
- 14h 16m
- Merged PRs (30d)
- 6
Description
### Summary
Each session's event log at `~/.copilot/session-state//events.jsonl` is appended one event at a time via `fsPromises.appendFile(path, line, { mode })` (guarded by a per-file `runExclusive` mutex). When `appendFile` is called with a **path** (rather than an open `FileHandle`), Node opens the file with flag `'a'`, writes, and **closes** it on every call — i.e. a `CreateFile → WriteFile → CloseHandle` cycle **per emitted event**.
On Windows, that `Create`+`Close` per event is what triggers Windows Defender (and other real-time AV) to rescan the file — the close, not the write, is the scan trigger. During an active, tool-heavy turn events are emitted tens–hundreds of times per second, so Defender rescans `events.jsonl` (which grows monotonically through the turn) that many times per second, producing a sustained CPU/energy regression on the user's machine.
We're seeing this downstream in a desktop app (Microsoft Scout) that bundles the Copilot CLI; `events.jsonl` is the largest remaining source of AV rescans after we cut our own state-file write churn. Tracked internally as ADO 62904486, and possibly the same root cause as #3907 ("CPU usage at max.").
### Where
- Append: `runExclusive(lockKey, async () => { await mkdir(dir, { recursive: true }); await appendFile(eventsPath, line, { mode }); })` — one open/append/close per event.
- Compaction: the `truncate` path does a single `writeFile(eventsPath, fullContent, { mode })` (full rewrite). That's occasional and fine.
### Ask
Hold a **persistent append `FileHandle`** open per session and reuse it:
- Open once per session with `fs.open(eventsPath, 'a')` (or a cached `createWriteStream`), then append via `fileHandle.appendFile(line)` / `fileHandle.write(line)` for each event, and `close()` once at session end (or on the `truncate`/compaction rewrite, then reopen).
- Because the handle stays open, the file identity is preserved and there is no per-event `Create`/`Close`, so AV real-time protection incrementally scans only the appended bytes instead of rescanning the whole file on every event.
- The existing `runExclusive` mutex already serializes writers, so a single long-lived handle is compatible — the only added lifecycle concern is closing/reopening around the `truncate` rewrite and on session teardown.
If a persistent handle is undesirable, a smaller mitigation is to **batch events** and flush on a short interval / at turn boundaries, which reduces (but doesn't eliminate) the open/append/close frequency.
### Impact
- Removes the dominant per-event AV rescan trigger on Windows; the cost currently scales with turn length × event volume, so long tool-heavy turns — when the machine is already busy — are the worst case.
- Append-only semantics and on-disk format are unchanged; `events.jsonl` remains the documented hook/event stream (cf. #3551).
### Environment
- OS: Windows 10/11, Microsoft Defender real-time protection enabled.
- Copilot CLI (`@github/copilot`) bundled inside an Electron desktop app.
- Path: `%USERPROFILE%\.copilot\session-state\\events.jsonl`.
- Observed mechanism: `fsPromises.appendFile(path, …)` per event (verified in the bundled `app.js`).
### Downstream reference
gim-home/m#4426 — same class of problem (Defender rescans from high-frequency small-file writes) for the desktop app's own `.scout` state files, mitigated there by coalescing writes; `events.jsonl` is the CLI-owned remainder.
Contributor guide
Research direction
Find the session event-log append implementation by searching for fsPromises.appendFile(eventsPath) and its runExclusive lock. Trace session teardown and the truncate or compaction path to understand when a handle must close or reopen. Done means events.jsonl keeps its existing format while appends reuse a persistent handle and lifecycle behavior remains safe.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100