openai / openai/codex

Official paginated resume drops the cumulative token ledger when TokenCount contains a decimal rate-limit percentage

Open
#38,116 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

app-server bug CLI
Dominant language
Rust
Stars
125k
Forks
19.5k
PR merge metrics
PR metrics pending

Description

What version of Codex CLI is running?

  • rust-v0.148.0-alpha.1

What subscription do you have?

  • ChatGPT Pro 20x

Which model were you using?

  • gpt-5.6-sol

What terminal emulator and version are you using (if applicable)?

  • Alacritty with Codex running inside tmux

Codex doctor report

Unavailable: all log recording is disabled for performance.

What issue are you seeing?

  • I discovered this after accidentally launching the official release against a long-running paginated thread that had previously survived multiple restarts with a local development build.
  • The last cumulative total before the official launch was 20,501,934,740 tokens. The first newly persisted total after official resume was 310,740, exactly the first request's usage rather than the previous total plus that usage.
  • Resume still loaded the conversation, but emitted no initial thread/tokenUsage/updated notification. Later totals continued from the corrupted near-zero baseline.
  • Returning to the local build prevented the same failure on later resumes but could not repair totals already written after the reset.

What steps can reproduce the bug?

  • Use an isolated CODEX_HOME containing a paginated rollout whose latest TokenCount has non-null info and a rate-limit window containing "used_percent":25.0.
  • Start codex app-server, initialize an experimental API client, and call thread/resume for that thread without supplying replacement history.
  • Observe that resume succeeds but emits no restored token-usage notification.
  • On a copy of the same rollout, change only the latest used_percent from JSON decimal 25.0 to the numerically equivalent JSON integer 25, leaving all token and other rate-limit fields unchanged.
  • Repeat the official resume. The persisted 63,048,929 cumulative total is now emitted immediately.

What is the expected behavior?

  • Every valid persisted TokenCount should deserialize during paginated reverse scanning, regardless of whether it includes rate-limit data.
  • Resume should seed the session with the latest persisted cumulative token usage, emit the restored usage notification, and continue accumulation from that value.

Additional information

  • codex-rs/exec-server-protocol/Cargo.toml enables the serde_json/arbitrary_precision feature for the binary dependency graph.
  • EventMsg is a tagged enum, while RateLimitWindow.used_percent uses the default f64 deserializer. Decimal values inside the buffered tagged payload are represented through serde_json's private number map, which the default f64 path rejects.
  • scan_model_context_from_lineage_blocking treats that RolloutLine as malformed and continues, so the enclosing TokenCount, including its otherwise valid info, is skipped. record_initial_history consequently finds no token ledger to restore.
  • The local development build happened to mask this upstream bug because an unrelated compatibility change for persisted rate-limit snapshots already deserialized used_percent through serde_json::Number and converted it with as_f64. That change was intended to make rate-limit percentages round-trip, but it also kept the enclosing TokenCount.info readable and therefore preserved the ledger.
  • A narrow upstream fix is the same field-level number conversion, with both a rollout round-trip test and a paginated-resume test that verifies cumulative token restoration when rate limits are present.
  • Current upstream main at 2230d64464488d8847197722fdca09d90095c705 retains both the arbitrary_precision feature and the unannotated f64 field.
  • This issue report was written with assistance from Codex CLI.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at RateLimitWindow.used_percent and trace scan_model_context_from_lineage_blocking through record_initial_history, focusing on paginated reverse scanning with arbitrary-precision JSON numbers. Add the narrow field-level deserialization fix described in the report, then add rollout round-trip and paginated-resume tests. Done means decimal rate-limit percentages deserialize, the latest cumulative TokenCount is restored, and resume emits the initial usage notification.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.