Investigation: dual-source session ingestion (chatSessions/ vs GitHub.copilot-chat/transcripts/) — dedup strategy needed before adopting transcript parser

Background

VS Code's Copilot Chat persists conversations in two different on-disk formats, and the extension currently only reads one of them. PR #70 proposes adding a parser for the second format, but in its current shape it causes duplicate sessions to appear in the Timeline (confirmed locally and in PR #70 comments by @eyehiel). Before merging any transcript-parsing code, we need to understand when each format is actually present, whether both are needed, and how to dedup.

This issue tracks that investigation. PR #70 is being closed in favor of this issue — the parser code there is a useful starting point but should not land until the questions below are answered.

The two formats

Format 1 — <workspaceStorage>/<hash>/chatSessions/*.{json,jsonl} (existing, parsed today)

• Purpose: Canonical persistent state for the Chat view — what VS Code reads back to restore/resume a conversation, show history, allow editing a past turn, etc.
• Shape: One file per session, structured JSON blob (whole-session snapshot, rewritten on update).
• Age: Original Copilot Chat persistence format; the extension has parsed this since day one.
• Scope: Per-workspace (lives under that workspace's workspaceStorage/<hash>/).

Format 2 — GitHub.copilot-chat/transcripts/*.jsonl (new, proposed in PR #70)

• Purpose: Agent Debug Logs event stream. Drives the Agent Debug panel (Preview) (Developer: Open Agent Debug Panel).
• Shape: Append-only NDJSON. One event per line: session.start, user.message, assistant.message, tool.execution_start, tool.execution_complete, plus system-prompt and customization events.
• Release history (per VS Code / Copilot Chat changelogs):
  • VS Code 1.110 / Copilot Chat 0.38 (Feb 2026) — Agent Debug panel (Preview) introduced.
  • VS Code 1.112 / Copilot Chat 0.40 (Mar 2026) — Export/import of agent debug logs.
  • VS Code 1.113 / Copilot Chat 0.41 (Mar 2026) — Agent debug logs for Copilot CLI and Claude CLI sessions.
• Scope: Can live at user-global path (e.g. ~/.vscode-server/data/User/GitHub.copilot-chat/transcripts/ on Remote/WSL) — i.e. not under the per-workspace workspaceStorage.
• Note: The chat UI does not read this back. It is a parallel side-channel for debug/audit/export.

Why this matters: PR #70 currently double-counts

PR #70 parses Format 2 and pushes the resulting sessions into the same sessions[] array as Format 1, with no dedup on sessionId. For any user where VS Code writes both (which appears to be the common case on recent versions), every conversation now appears twice in the Timeline.

Confirmed locally:

• Build from main (Format 1 only) → one entry per session.
• Build from PR #70 head → two entries per session (same id, slightly different field coverage).

Reproduces what @eyehiel reported on PR #70.

Open questions to answer before merging

1. Coverage: Is Format 1 (chatSessions/) effectively guaranteed for every interactive Copilot Chat / agent session on supported VS Code versions? If so, Format 2 should be treated as enrichment, not as a primary source.
2. Format 1-missing edge cases: When can a session legitimately exist in Format 2 but not Format 1? Candidates to verify:
   • Sessions created via the Agent Debug panel itself.
   • Copilot CLI / Claude CLI sessions logged via debug-logs (v1.113+).
   • Sessions whose chatSessions/ blob was pruned (history limits, "Clear All Chats", deleted workspace) while the global transcripts/.jsonl survives.
   • Pre-/post-version mismatches across VS Code Stable, Insiders, and Remote (.vscode-server).
3. Field richness: What does Format 2 contain that Format 1 does not (e.g. precise tool start/complete timestamps, system-prompt events, customization-load events)? Is any of that data worth merging into the Format 1 session when both exist?
4. Same sessionId guarantee: Do both formats consistently use the same session identifier for the same conversation? (Spot-check across Stable, Insiders, Remote-WSL, Remote-SSH, devcontainer.)
5. Path discovery for Format 2: Transcripts are user-global, not per-workspace. How should we enumerate them on:
   • Windows desktop (%APPDATA%\Code\User\GitHub.copilot-chat\transcripts\)?
   • macOS / Linux desktop (~/.config/Code/User/...)?
   • VS Code Server / WSL / Remote-SSH / devcontainer (~/.vscode-server/data/User/...)?
   • Insiders variants (Code - Insiders, .vscode-server-insiders).
6. Retention / size: Are append-only .jsonl transcripts subject to any retention policy? Do we need to cap file size or age when ingesting?

Proposed direction (to validate during investigation)

• Primary: keep chatSessions/*.{json,jsonl} as the canonical source.
• Secondary: read transcripts/*.jsonl only to (a) recover sessions absent from Format 1, and (b) enrich existing sessions (extra tool/system-prompt events) — keyed by sessionId.
• Hard rule: never emit two Session objects for the same sessionId from two sources.

Related work

• PR #63 / Issue #62 — .vscode-server workspaceStorage discovery for WSL / Remote-SSH / devcontainer (already-landed dependency for any remote-format work).
• PR #70 — initial transcript parser implementation (being closed in favor of this issue).
• Issue #64 — original feature request for transcripts/ parsing.

Definition of done for the investigation

• Documented answers to all six questions above with reproduction notes (which VS Code build, which host environment).
• Decision recorded on: single-source vs dual-source with dedup vs dual-source with enrichment-merge.
• Updated parser proposal that cannot regress users currently on Format 1 only.
• Test coverage that explicitly asserts no sessionId appears twice across sources, on both Windows and non-Windows hosts (the failing test in PR #70 around findVsCodeDirs should also be addressed: it must be skipped on Windows or the production guard reworked).

cc @hora7ce @eyehiel — thank you both, the PR + screenshot were exactly what surfaced this. Help validating the edge cases on WSL / Remote setups would be very welcome.