pingdotgg / pingdotgg/t3code

[Bug]: Codex usage parser does not reconcile per-event and cumulative counters

Open
#5,800 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
23k
Forks
5.9k
Avg merge
11h 14m
Merged PRs (30d)
357

Description

Before submitting
  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.
Area

apps/server

Steps to reproduce
  1. Create a Codex rollout containing a turn_context followed by a valid token_count event whose total_token_usage advances by its last_token_usage.
  2. Add another token_count event where the cumulative total_token_usage is unchanged but last_token_usage differs, as can happen when Codex recomputes current-context usage or re-emits token state alongside rate-limit updates.
  3. Alternatively, make total_token_usage advance by a value that does not equal last_token_usage.
  4. Scan the rollout from the Usage page.

The Codex protocol maintains total_token_usage by adding each real last_token_usage element-wise. An unchanged cumulative total therefore proves that a later event did not add billable usage, and a different cumulative delta proves that the two counters are inconsistent.

Expected behavior

The usage parser should use total_token_usage as an integrity signal. It should count a legitimate per-request last_token_usage once, suppress events whose cumulative total did not advance, and report inconsistent counter sequences as malformed rather than guessing which counter is correct.

Reasoning tokens must remain a subset of output tokens and must not be added to the processed-token total again.

Actual behavior

parseCodexLine never reads total_token_usage. It trusts each nonzero last_token_usage whose JSON signature differs from the immediately previous one.

This means a stale or recomputed last_token_usage can be counted even when the cumulative total did not advance, and a malformed per-event counter can be accepted even when it cannot reconcile with the cumulative counter. The Usage feature's original PR stated that these fields had been reconciled before last_token_usage was trusted, but that invariant is not implemented in the parser.

This is separate from copied parent history in forked rollout files (#5758): the issue here occurs within one rollout's counter sequence.

Impact

Major degradation or frequent failure

Version or commit

main @ 963ebf5bd7cce00d40ff60c258b34c12dcab271e

Environment

Platform independent; Codex rollout transcript parsing in apps/server

Logs or stack traces
No error is logged. Inconsistent token counters are accepted silently.
Screenshots, recordings, or supporting files

No response

Workaround

Use a usage parser that validates each per-event counter against the cumulative Codex counter.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at parseCodexLine in apps/server and trace how token_count events are scanned for the Usage page. Compare each last_token_usage value with total_token_usage across the rollout, including unchanged and inconsistent cumulative counters. Done means valid usage is counted once, non-advancing events are suppressed, malformed sequences are reported, and reasoning tokens are not counted twice.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.