Paginated history drops valid flattened rollout records and reuses ordinals

Open
#35,746 42 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust, sqlite
Domain
backend, cli, databases

Research direction

Start with scan_next::RolloutLine in codex-rs/rollout/src/ordinal.rs and the line deserialization in codex-rs/thread-store/src/local/thread_history_materialization.rs. Reproduce the provided token-count serialization case, then trace resume ordinal discovery and SQLite projection catch-up. Done means both readers match the canonical value-first decoding semantics and regression coverage confirms valid final records and affected projections are handled correctly.

Written by the indexing model from the issue text.

Description

bug CLI session
What version of Codex CLI is running?

Observed on 0.146.0-alpha.10.1. The affected source paths remain unchanged in rust-v0.146.0-alpha.14.

What platform is your computer?

Linux x86_64.

What issue are you seeing?

Paginated rollout history has inconsistent RolloutLine decoding. Some readers deserialize JSON directly into RolloutLine, while the canonical loader first parses into serde_json::Value and then calls serde_json::from_value::<RolloutLine>.

A serialized RolloutLine::EventMsg(EventMsg::TokenCount(...)) containing populated rate-limit and credit data is valid JSON and succeeds through the value-first path, but direct serde_json::from_str::<RolloutLine> / serde_json::from_slice::<RolloutLine> rejects it.

Two paginated-history readers currently use the direct path:

  • codex-rs/rollout/src/ordinal.rs: scan_next::<RolloutLine>()
  • codex-rs/thread-store/src/local/thread_history_materialization.rs: serde_json::from_slice(line_bytes)

This produces two related failures:

  1. Resume ordinal discovery silently skips the valid final record, steps back to an earlier ordinal, and can append a new record with an already-used ordinal.
  2. SQLite history projection silently skips the valid record while advancing its byte offset beyond it, leaving the materialized projection behind the canonical JSONL permanently.

Once affected, catch-up can begin with one replayed boundary ordinal equal to projection_state.next_ordinal - 1; rejecting that boundary prevents the remaining valid suffix from materializing.

What steps can reproduce the bug?

Add a focused test that serializes a paginated RolloutLine containing a token-count event with rate-limit and credit fields:

let encoded = serde_json::to_string(&RolloutLine {
    timestamp: "2026-07-09T00:00:05Z".to_string(),
    ordinal: Some(5),
    item: RolloutItem::EventMsg(EventMsg::TokenCount(TokenCountEvent {
        info: Some(TokenUsageInfo::full_context_window(1_000)),
        rate_limits: Some(RateLimitSnapshot {
            limit_id: Some("limit-1".to_string()),
            limit_name: None,
            primary: Some(RateLimitWindow {
                used_percent: 1.0,
                window_minutes: Some(60),
                resets_at: Some(1),
            }),
            secondary: None,
            credits: Some(CreditsSnapshot {
                has_credits: true,
                unlimited: false,
                balance: Some("1".to_string()),
            }),
            individual_limit: None,
            spend_control_reached: None,
            plan_type: None,
            rate_limit_reached_type: None,
        }),
    })),
})?;

assert!(serde_json::from_str::<RolloutLine>(&encoded).is_err());
assert!(
    serde_json::from_str(&encoded)
        .and_then(serde_json::from_value::<RolloutLine>)
        .is_ok()
);

Then:

  1. Append that record as the final line of a paginated rollout and resume it. Ordinal discovery ignores ordinal 5.
  2. Materialize the same rollout into the thread-history SQLite projection. The line is rejected, but the stored byte offset advances past it.
What is the expected behavior?

Every rollout reader should use the same value-first decoding semantics as the canonical loader. Resume should continue after the real final ordinal, and SQLite materialization should project every valid canonical record.

For projections already affected by the old decoder, narrowly accepting and skipping one replayed first boundary ordinal (next_ordinal - 1) allows the remaining suffix to catch up without rewriting canonical JSONL.

Additional information

rust-v0.146.0-alpha.14 changes nearby thread-list behavior for missing rollout paths, but ordinal.rs and thread_history_materialization.rs still use direct RolloutLine deserialization.

Dominant language
Rust
Stars
125k
Forks
19.5k
Avg merge
1m
Merged PRs (30d)
1k

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from openai/codex

All issues in openai/codex

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.