Session cost total silently loses a fixed chunk of spend the first time context compaction succeeds in a process's lifetime
- Lingua principale
- Shell
- Stelle
- 11.2k
- Fork
- 1.9k
- Merge medio
- 14h 16m
- PR unite (30g)
- 6
Descrizione
### Describe the bug
**Environment:** Copilot CLI v1.0.77, macOS, `hostType: github`
**Scope:** This has only been observed and verified in the Copilot CLI. I
have not tested or verified whether this affects other Copilot clients/IDEs
(VS Code extension, JetBrains, web, etc.) — the mechanism described below is
specific to the CLI's local session/event architecture (`events.jsonl`,
`session.compaction_*` events, local SQLite usage ledger), which may or may
not be shared with other clients.
**Summary:** In a long-running, tool-call-heavy session, the exit banner
reported **508 AI Credits**. Summing every individual request logged for the
same session in the local usage ledger (`assistant_usage_events` in the
CLI's local SQLite store) totals **873.81 AI Credits** across 158 requests —
a **365.46 AIC undercount**. Critically, **the local ledger's total is the
one that matches actual billed usage** (confirmed against the increase in
account usage shown in the GitHub billing/web UI for this period) — **the
exit banner and live in-session indicator are the ones that are wrong**, not
the other way around. Continued investigation (resuming the session,
closing it again, and analyzing a second shutdown) pinpointed the exact
mechanism: **a fixed, one-time chunk of already-accrued-but-unflushed cost
disappears at the moment of the *first successful context-compaction* in a
given CLI process's lifetime** (whether that's the original session launch
or a later `/resume`). Before that first successful compaction, tracking is
invisible to us; after it, tracking is essentially perfect (drift-free to
within rounding) until the process ends or is resumed again, at which point
the cycle repeats.
#### Reproduction / evidence
**1. The gap is a fixed dollar amount, not a percentage — confirmed across 8 readings:**
| Reading | Time | Shutdown/checkpoint reported | True ledger total | Gap (AIC) |
|---|---|---|---|---|
| checkpoint 1 | 19:43:31 | 350.55 | 716.01 | 365.46 |
| checkpoint 2 | 19:45:24 | 402.08 | 767.55 | 365.47 |
| checkpoint 3 | 19:49:34 | 482.45 | 847.91 | 365.46 |
| checkpoint 4 | 19:52:36 | 499.81 | 865.27 | 365.46 |
| checkpoint 5 | 19:53:31 | 505.10 | 870.56 | 365.46 |
| **shutdown 1** | 20:06:26 | 508.34 | 873.81 | 365.47 |
| checkpoint 7 (post-resume) | 20:21:00 | 593.97 | 1,033.07 | 439.10 |
| **shutdown 2** (post-resume, final) | 20:25:58 | 636.07 | 1,075.16 | 439.09 |
The gap holds *exactly flat* at 365.46-365.47 AIC across 6 consecutive
readings spanning 22+ minutes and 331.78 AIC of brand-new, correctly-tracked
activity — then jumps once to a new fixed value (439.09-439.10) and holds
flat there too.
**2. Both jumps land within 0.5 seconds of a compaction event:**
- Jump 1 (+365.46 AIC): the cumulative ledger crosses this threshold at
**19:24:19.7Z** — 0.5s before `session.compaction_complete` fires at
19:24:20.2Z. This is the session's very **first** compaction attempt.
- Jump 2 (+73.63 AIC): the cumulative ledger crosses this threshold at
**20:15:45.5Z** — 0.5s before the **next successful** `session.compaction_start`
fires at 20:15:46.1Z.
**3. The critical detail that makes the pattern click: one compaction attempt failed.**
The session went through 4 compaction *attempts* total, but only 3 succeeded:
| # | Start | Result | Context |
|---|---|---|---|
| 1 | 19:22:38 | ✅ success | 1st compaction of the **original process launch** — **LOST 365.46 AIC** |
| 2 | 19:45:06 | ✅ success | 2nd compaction of the same process launch — clean, no loss |
| 3 | 20:15:04 | ❌ **failed** — `"error": "Compaction failed: received empty response from model"` | 1st compaction attempt after `/resume` — never completed, no state change |
| 4 | 20:15:46 | ✅ success | Retry, 30s later — effectively the **1st successful compaction after resume** — **LOST 73.63 AIC** |
Excluding the failed attempt, **every loss event corresponds exactly to the
first successful compaction of a process's lifetime** — the original launch
(`session.start` at 19:02:57) and the later `/resume` (confirmed via a
`session.resume` event at 20:12:44.539Z) each produced exactly one loss, at
their respective first successful compaction, and no loss on any subsequent
compaction in that same process's lifetime.
#### Root cause (strong hypothesis, needs engineering confirmation)
The in-memory cost/usage accumulator appears to be (re)initialized whenever
the CLI process starts or a session is resumed. It tracks new activity
correctly from that point — but the **first time a compaction successfully
completes** in that process's lifetime, whatever internal state-reconciliation
or checkpoint-numbering logic compaction performs mishandles the handoff,
**silently discarding the cost accrued since process start that hadn't yet
been captured by a periodic flush**. Because this reconciliation only
misfires once per process lifetime, later compactions in the same run are
unaffected — but each new process attach (via `/resume`, or possibly a crash
recovery) resets the clock and makes the *next* first-successful-compaction
vulnerable again.
#### Impact
Any session that is long/complex enough to trigger auto-compaction will
under-report real spend by whatever cost had silently accrued before that
first successful compaction — and if the session is resumed later, this can
happen a second time. For very long sessions with multiple resumes, this
loss could compound to a significant, unbounded amount over time. This is
not just a cosmetic display bug: users relying on the live in-session
indicator or the exit-banner total for budgeting will be materially misled
about how much they were actually charged, since the local ledger — not
the banner — is what matches real billed usage.
#### Suggested fix direction
Audit the compaction success-path for whatever step transitions/reconciles
the cost accumulator (likely tied to the `checkpointNumber` assignment logic,
since successful compactions carry a `checkpointNumber` and failed ones do
not). Ensure it merges with the accumulator's current value instead of
overwriting/reinitializing it. Also worth checking whether the periodic
`session.usage_checkpoint` mechanism only begins firing *after* the first
compaction in a process lifetime (it appears to start only around then in
this session), which would independently explain why the first compaction
window has no earlier flush to protect it.
#### Appendix: independently reproduced in a second, unrelated session
While preparing this report (itself running in a separate CLI session), that
session hit its own first auto-compaction. The same signature reproduced
exactly:
- All 12 periodic checkpoints before the first compaction matched the
authoritative per-request ledger with **zero gap**.
- The instant `session.compaction_complete` fired for that session's first
(and so far only) compaction, the ledger jumped by ~48.7 AIC in a single
request-processing gap, and the next periodic checkpoint showed a **~30.9
AIC gap** versus the ledger that had not existed moments before.
This is a second, independent data point (different repo, different
conversation, no relationship to the session in the main report above)
showing the loss occurring at the exact same trigger — the first successful
compaction of a process's lifetime — reinforcing that this is a general,
reproducible mechanism rather than something specific to one unusual
session.
#### Appendix: secondary, distinct issue found during investigation
Separately from the cost-accounting bug above: **one conversational turn in
this session has zero rows in the per-request usage ledger** despite being a
real turn with a real assistant response (the ledger's turn-index sequence
jumps directly over it). This is data missing at the recording layer itself,
not merely excluded from a summary — a different failure mode from the
cost-accumulator bug above, and possibly worth a separate, smaller bug report
if useful.
### Affected version
GitHub Copilot CLI 1.0.77.
### Steps to reproduce the behavior
1. Start a Copilot CLI session and work in it long enough (many tool calls /
agentic turns) to trigger the CLI's automatic context compaction (occurs
around the ~200k token threshold — watch for a `/compact`-style event or
just a long, tool-heavy session).
2. Before that first compaction, periodically note the AI Credits total
shown live in the CLI (or check `session.usage_checkpoint` events in that
session's `~/.copilot/session-state//events.jsonl`) — this
matches actual usage exactly at this stage.
3. Let the first auto-compaction complete successfully (a
`session.compaction_complete` event with `"success": true` in the same
`events.jsonl`).
4. Compare the live/reported AI Credits total immediately after that
compaction against the sum of `total_nano_aiu` for that session in the
local usage ledger (`~/.copilot/session-store.db`, table
`assistant_usage_events`, `SELECT SUM(total_nano_aiu) FROM
assistant_usage_events WHERE session_id = ''`, divided by 1e9 to get
AI Credits). A gap appears at this exact moment and does not go away.
5. Continue working and close the session; compare the final exit-banner
total against the same ledger `SUM()` query. The exit banner is now a
fixed number of credits lower than the actual/billed total, matching the
gap that first appeared at step 3-4.
6. (Optional, to fully confirm the mechanism) If the session is later
resumed and undergoes another *successful* compaction, watch whether
another fixed-size gap appears at that point too — this happened in both
sessions I tested it in (once per session per process attach: original
launch and again after resume).
I don't have a single CLI command that reproduces this in isolation (it
depends on the CLI's internal auto-compaction trigger) — this was found and
confirmed via the two data sources above (live/exit-banner totals vs. the
local SQLite ledger), cross-referenced against each session's own
`events.jsonl` compaction event timestamps.
### Expected behavior
The AI Credits total shown live during a session, and the final total shown
in the exit banner, should always match the actual usage that gets billed
to the account (i.e., match the sum of the per-request local usage ledger).
Successfully completing a context-compaction should never cause any
already-accrued cost to be silently dropped from the running/reported total
— compaction should only affect the conversation context sent to the model,
not the accounting of prior usage.
### Additional context
- The local per-request usage ledger (`assistant_usage_events` in
`~/.copilot/session-store.db`) appears to be the authoritative source —
its totals match the actual increase in account usage shown in the
billing/usage UI. The live in-session indicator and exit-banner total are
the values that are wrong.
- The loss is a **fixed, one-time amount**, not a percentage or an ongoing
drift — it appears exactly once per successful compaction that is the
*first* successful compaction of a given CLI process's lifetime (whether
that's the original session launch, or after a `/resume`). Later
compactions within that same process lifetime do not cause further loss.
- I verified this pattern independently in two separate sessions (different
repositories, unrelated conversations), with the same signature both
times: zero gap before the first successful compaction, then a fixed gap
appearing right at that compaction's timestamp and persisting afterward.
- This has only been observed/tested in the Copilot CLI — I have not
verified whether it affects other Copilot clients (VS Code extension,
JetBrains, web, etc.).
- Separately, I also noticed the local usage ledger itself is occasionally
missing an entire conversational turn's usage rows (a turn with a real
assistant reply but zero corresponding ledger entries) — this looks like a
distinct, second data-loss mechanism, unrelated to the compaction-timing
issue described above, and may be worth investigating separately.
- Happy to share the exact session ID and further supporting detail via the
private feedback channel if useful for your team to pull internal logs.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia confrontando session-state//events.jsonl con la query assistant_usage_events in ~/.copilot/session-store.db in corrispondenza del primo evento session.compaction_complete riuscito. Esamina il percorso di successo della compaction, la gestione di checkpointNumber e il comportamento di session.usage_checkpoint descritto nel report. Il lavoro è completato quando i totali live e dell'exit-banner continuano a corrispondere al ledger dopo la compaction iniziale e dopo la compaction attivata da resume.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- shell, sqlite
- Ambito
- cli, databases
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 45/100