Codex checkpoint hooks slow down over a session (O(N²) transcript reparse in token-usage calc)

This was generated by AI during triage.

Summary

Codex checkpoint hooks get progressively slower over a session and visibly "hang" at
turn end, while Claude Code hooks stay fast on comparable sessions. Root cause: on
every turn-end (Stop) hook the Codex agent JSON-parses its entire cumulative
rollout transcript to compute token usage, whereas the Claude Code agent slices the
transcript first and parses only the lines added since the last checkpoint. Codex's
per-hook cost is therefore O(session length), making the whole-session cost O(N²); the
larger byte size of Codex rollouts (they embed encrypted_content reasoning blobs)
multiplies the constant factor.

This issue is the entry point for an implementation agent. The analysis below is the
context; the Agent Brief section is the contract.

---

Background: how the hook pipeline works

All agents share one turn-end hook pipeline (the lifecycle dispatcher's TurnEnd
path). Per hook it: prepares the transcript (optional, agent-specific), reads the whole
transcript file into memory, copies it into the session metadata dir, extracts modified
files, computes token usage, and calls the strategy's SaveStep. The read, the copy,
and the token/file parsing are all size-linear in the transcript, and they run every
turn. So per-hook latency is dominated by transcript size and by how much of the
transcript each step reprocesses.

Root cause (the divergence)

Token usage is computed through the agent's TokenCalculator /
SubagentAwareExtractor interface, given the full in-memory transcript bytes plus a
fromOffset marking where the current checkpoint's window starts.

• Claude Code slices first — it uses the shared transcript.SliceFromLine helper to
  cut the byte buffer down to the current checkpoint window, then runs
  transcript.ParseFromBytes on that slice. JSON unmarshalling touches only the new
  lines since the last checkpoint. Per-session parse cost is O(N).

• Codex does not slice. CodexAgent.CalculateTokenUsage splits the entire rollout
  into lines (an internal splitJSONL-style whole-file split) and json.Unmarshals
  every line from session start to check its envelope type; fromOffset only
  decides which side of the token delta an already-parsed line contributes to. It is
  forced to start from line 0 because Codex reports cumulative token counts
  (event_msg → token_count → total_token_usage): computing this turn's delta needs
  the last cumulative value at/before fromOffset (the baseline) and the last value
  after it. So every hook reparses the whole growing file. Per-session parse cost is
  O(N²), and late-turn hooks feel like a hang.

Two smaller contributors, same subsystem:

• Codex has no bytes-based subagent-aware extractor, so modified-file extraction takes
  the fallback path and re-opens and re-reads the rollout file from disk
  (ExtractModifiedFilesFromOffset(path, ...)), a second full-file read on top of the
  in-memory copy the pipeline already holds.
• Codex rollout lines carry encrypted_content reasoning blobs (large base64), so each
  size-linear step processes many more bytes than a Claude transcript for equivalent
  work. (The Codex transcript sanitizer already strips these blobs on the
  restore/portability path — evidence they are present and large.)

Ruled out (do not chase these)

• The Codex PostToolUse per-edit hook is already lightweight (records touched files
  from the hook's stdin payload; no SaveStep, no transcript parse). It is not the
  cause.
• Claude Code's PrepareTranscript flush-wait is a Claude-only cost, is capped and
  tail-only, and would only ever make Claude slower — Codex does not run it.
• GetTranscriptPosition is a cheap byte-scan line count in both agents (no JSON).

---

Agent Brief

Category: enhancement (performance)

Summary: Make the Codex agent's turn-end token-usage computation incremental so hook
latency stops scaling with total session length.

Current behavior:
On every turn-end hook, the Codex agent's TokenCalculator implementation parses the
entire rollout transcript from line 0, regardless of fromOffset. Because the parse is
repeated each turn and the rollout grows every turn, checkpoint-hook latency grows
roughly linearly within a session and the whole-session cost is quadratic. Large
encrypted_content reasoning payloads in Codex rollouts amplify the constant factor. In
long sessions this manifests as a multi-second stall at the end of each turn.

Desired behavior:
Codex token-usage computation at a checkpoint should cost O(new transcript content since
the last checkpoint), not O(whole session), while producing identical token numbers to
today (the same per-checkpoint delta and the same session totals used by
entire status, entire checkpoint tokens, etc.). The fix must preserve Codex's
cumulative-counter semantics: the delta for a checkpoint is
last_total_after_offset − last_total_at_or_before_offset, with the pre-offset baseline
still available even though the pre-offset lines are no longer re-parsed on the hot path.

Suggested approach (agent decides the actual design): persist the last observed
cumulative total_token_usage snapshot in the per-session state that already tracks
checkpoint token accounting (the session state carries TokenUsage and
CheckpointTokenUsage), then on the next checkpoint parse only the lines after
fromOffset (via the shared transcript.SliceFromLine helper or an equivalent
byte-offset slice) and read the baseline from state instead of rescanning history.
Handle the cold-start case (no persisted baseline yet — e.g. first checkpoint, resumed
or imported session) by falling back to the existing full scan for that one hook.

While in this code path, also give Codex a bytes-based modified-file extractor so
turn-end file extraction reuses the transcript bytes already held in memory instead of
re-reading the rollout file from disk (mirror the pattern the Claude Code and Factory AI
Droid agents use for their *FromBytes extractors / SubagentAwareExtractor). Codex has
no subagents, so no subagent-transcript discovery is required — just avoid the second
full-file disk read.

Key interfaces:

• CodexAgent.CalculateTokenUsage(transcriptData []byte, fromOffset int) — must become
  incremental; must return the same agent.TokenUsage (fresh input, cache-read, output,
  API-call count) it returns today for any given session state.
• The agent.TokenCalculator contract and, if a bytes-based file extractor is added,
  agent.SubagentAwareExtractor (ExtractAllModifiedFiles(data, fromOffset, subagentsDir))
  — Codex may implement the latter to route file extraction through in-memory bytes.
• The strategy SessionState (fields TokenUsage, CheckpointTokenUsage) as the place
  to persist the cumulative baseline across hooks — reuse existing accounting rather than
  adding a parallel store if practical.
• Codex rollout envelope types (rolloutLine, event_msg / token_count /
  total_token_usage) — the parse still keys off these; only the span of lines parsed
  changes.

Acceptance criteria:

• For a session of N turns, the number of transcript lines JSON-parsed by the Codex
  token-usage path on a single turn-end hook is bounded by the lines added that turn
  (plus O(1) baseline lookup), not by total session length. Demonstrated by a unit
  test that asserts parse work scales with the delta, not the whole transcript.
• Per-checkpoint and cumulative token numbers are unchanged versus the current
  implementation for representative rollout fixtures (add/keep fixtures covering
  multiple turns, cached-input tokens, and a resumed/mid-session offset).
• Cold start (no persisted baseline) still produces correct totals via a one-time
  fallback; a test covers first-checkpoint and imported/resumed-session cases.
• Turn-end modified-file extraction for Codex no longer performs a second full-file
  disk read of the rollout (extraction runs on the in-memory transcript bytes).
• Existing Codex agent tests pass; new tests use t.Parallel() and the testutil
  isolated-repo helpers per repo conventions.
• mise run check passes (fmt, lint, unit + integration tests).

Out of scope:

• The unconditional whole-file transcript copy into the session metadata dir on
  every hook — it affects all agents and is a separate optimization. Do not change the
  shared lifecycle pipeline here.
• Stripping encrypted_content from stored checkpoints to shrink transcripts — separate
  concern with its own restore/portability implications; track separately.
• Any change to Claude Code, other agents, or the shared transcript package's public
  behavior beyond additive helpers.
• Changing token-accounting semantics, output fields, or how entire status /
  checkpoint tokens report numbers.

---

How to verify the win empirically

The pipeline is instrumented with perf spans. Enable debug/perf logging by adding
"log_level": "DEBUG" to .entire/settings.local.json (gitignored) or exporting
ENTIRE_LOG_LEVEL=debug, run comparable Codex sessions before/after, and compare the
extract_metadata span (token calc + file extraction) across turns — today it grows
super-linearly with turn count for Codex; after the fix it should stay roughly flat per
turn. entire doctor trace surfaces these spans.

Orienting pointers (may drift — rely on the names above, not these)

• Codex agent: cmd/entire/cli/agent/codex/ (transcript.go, lifecycle.go).
• Claude Code agent for the reference incremental pattern:
  cmd/entire/cli/agent/claudecode/transcript.go (CalculateTotalTokenUsage).
• Shared slicing/parsing: cmd/entire/cli/transcript/ (SliceFromLine, ParseFromBytes).
• Shared hook pipeline: cmd/entire/cli/lifecycle.go (turn-end path).

Start in cmd/entire/cli/agent/codex/transcript.go and lifecycle.go, then compare Claude Code's incremental pattern in cmd/entire/cli/agent/claudecode/transcript.go and the shared SliceFromLine and ParseFromBytes helpers. Trace SessionState token accounting and the turn-end path in cmd/entire/cli/lifecycle.go before locating Codex tests. Done means incremental token parsing preserves totals, cold starts work, and modified-file extraction uses in-memory bytes without a second disk read.