applyBackfilledSessionTokenUsage overwrites the session-wide token total with the checkpoint window delta for every non-Copilot agent
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 475
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 178
Description
SessionState.TokenUsage is documented as "the session-wide total used by entire status", with SessionState.CheckpointTokenUsage as the per-window delta (CLAUDE.md:1850). resetCheckpointWindow clears the latter and deliberately leaves the former alone, confirming the intent. But every condensation overwrites TokenUsage with the window delta anyway, so after the first commit entire status and entire session tokens under-report the session by everything that happened before it.
Mechanism
CondenseSession calls applyBackfilledSessionTokenUsage(ctx, ag, state, sessionData.Transcript, sessionData.TokenUsage) (cmd/entire/cli/strategy/manual_commit_condensation.go:561), where sessionData.TokenUsage is scoped to state.CheckpointTranscriptStart — the current window, not the session.
sessionStateBackfillTokenUsage exists to recover a full-session total for Copilot CLI, which writes session.shutdown after the hooks return. Its first two branches are Copilot-only and correct. The third is not gated by agent type at all:
if checkpointUsage != nil && checkpointUsage.InputTokens > 0 {
return checkpointUsage
}
cmd/entire/cli/strategy/manual_commit_condensation.go:1193
So for Claude Code, Codex, Cursor, Gemini, opencode, Pi and Factory Droid, any window with input tokens returns the window value, and applyBackfilledSessionTokenUsage (line 1175) assigns it straight onto state.TokenUsage. Only SubagentTokens is preserved across the assignment (replaceSubagentTokensFrom) — the main fields are not.
The effect compounds rather than self-correcting: SaveStep accumulates onto state.TokenUsage (cmd/entire/cli/strategy/manual_commit_git.go:124), so each subsequent turn builds on the reduced base. A session with many commits reports roughly "tokens since the last commit" indefinitely.
Reproduction
state := &SessionState{
SessionID: "s1", AgentType: agent.AgentTypeClaudeCode,
TokenUsage: &agent.TokenUsage{InputTokens: 10000, OutputTokens: 999, APICallCount: 42},
}
window := &agent.TokenUsage{InputTokens: 100, OutputTokens: 10, APICallCount: 1}
applyBackfilledSessionTokenUsage(t.Context(), nil, state, nil, window)
// state.TokenUsage is now {InputTokens: 100, OutputTokens: 10, APICallCount: 1}
Verified against origin/main: the session total goes 10000 to 100.
Affected output
entire status (cmd/entire/cli/status.go:642, via totalTokens(st.TokenUsage)) and entire session tokens (cmd/entire/cli/session_tokens.go:208, via buildSessionTokensUsage(state.TokenUsage)). Checkpoint metadata is unaffected — it is supposed to carry the window delta and still does.
Suggested direction
The third branch looks like it was meant as the Copilot fallback and lost its agent-type guard; the branch directly above it is the same fallback, gated. The narrow fix is to scope it to Copilot CLI and return nil otherwise, leaving state.TokenUsage as the hook-accumulated cumulative for every other agent.
That needs a check on whether any agent currently depends on the unguarded branch to get a total at all — an agent whose hooks report no token usage would go from "window delta" to "nothing", which may be the better answer but is a visible change either way.
Worth a test pinning that a non-Copilot session's TokenUsage survives condensation, since nothing currently covers it.
Provenance
Found while reviewing #2334, which touched applyBackfilledSessionTokenUsage but did not introduce this — the unguarded branch and sessionStateBackfillTokenUsage predate it. Filed separately as agreed in that review.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in cmd/entire/cli/strategy/manual_commit_condensation.go, reading applyBackfilledSessionTokenUsage and sessionStateBackfillTokenUsage, then inspect the CondenseSession call at line 561 and SaveStep in manual_commit_git.go. Add or update a focused test showing that a non-Copilot session retains its cumulative TokenUsage after condensation, while preserving the intended Copilot behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100