openai / openai/codex

# Bug report: Codex session history disappears when durable-rollout projection rejects `token_count` events

Open
#42,025 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

app bug remote session
Dominant language
Rust
Stars
125k
Forks
19.5k
PR merge metrics
PR metrics pending

Description

What version of Codex CLI is running?

codex-cli 0.147.0

What subscription do you have?

Pro plan

Which model were you using?

Sol and luna

What platform is your computer?

Darwin 25.3.0 arm64 arm

What terminal emulator and version are you using (if applicable)?

cmux

Codex doctor report

What issue are you seeing?

Bug report: Codex session history disappears when durable-rollout projection rejects token_count events

Product: Codex (desktop/remote SSH workflow and Codex CLI)

Severity: High — active work remains on disk, but the session cannot be reliably discovered or viewed through the normal history/sidebar UI.

Date observed: 2026-09-01 (Asia/Shanghai)

Summary

Codex sessions on a remote Linux host (“D2”) intermittently disappear from the laptop/sidebar, and the current session’s history becomes incomplete. The underlying rollout JSONL files are still present and valid, so this does not appear to be filesystem deletion. The local thread-history projection stops processing new rollout events after a token_count record and repeatedly logs a type/ordinal mismatch.

The same projection error has been observed on at least ten threads since 2026-08-30 21:53 (Asia/Shanghai).

Affected environment

  • Remote host: D2, accessed from Codex desktop on a laptop through the SSH remote connection.
  • Repository: /home/dev/projects/agentic-skills/super-agent
  • Codex CLI installed on the remote host: 0.145.0
  • Local Codex state directory: /home/dev/.codex
  • State databases: state_5.sqlite and thread_history_1.sqlite
  • Current affected thread source: vscode / user-created thread

Reproduction / observed sequence

  1. Start or resume a Codex session on the D2 remote connection in the repository above.
  2. Allow the session to run a workflow with tool calls and normal token-usage updates.
  3. Disconnect/reconnect the laptop or reopen Codex.
  4. Search for the session by its title, project, or branch.
  5. The session may be absent from the sidebar/search, or may reopen with history stopping before the latest turns.
  6. Resume by the exact UUID. The rollout file continues to contain the conversation, but the UI projection remains stuck.

Concrete evidence

Affected current thread:

UUID: 01a05aa0-e681-7853-8054-0db7fe82ac89
Archived: false
CWD: /home/dev/projects/agentic-skills/super-agent
Rollout: /home/dev/.codex/sessions/2026/09/01/rollout-2026-09-01T09-41-29-01a05aa0-e681-7853-8054-0db7fe82ac89.jsonl

Read-only inspection found:

  • Rollout file size: approximately 3.7 MB.
  • Rollout file records: 355; all records parse as valid JSON.
  • Latest rollout activity was still present through 2026-09-01 16:24:10 (Asia/Shanghai).
  • thread_history_projection_state.next_rollout_byte_offset is stuck at ordinal 17.
  • thread_turns contains one turn with status inProgress.

Relevant log messages:

failed to project durable rollout for 01a05aa0-e681-7853-8054-0db7fe82ac89:
thread-store internal error: thread history projection ... expected ordinal 17, got 18

skipping rejected rollout line while projecting ...:
invalid type: map, expected f64 at line 1 column 806

The rejected record is a token_count event. Its payload contains structured fields including last_token_usage, total_token_usage, model_context_window, and a structured rate_limits.credits object. This suggests a materializer/parser schema mismatch, although the exact field causing the f64 expectation should be confirmed by OpenAI.

The same invalid type: map, expected f64 projection error appears repeatedly for multiple threads. The D2 filesystem has ample free space, and the rollout files remain intact.

Affected thread IDs found in the local log

The following ten thread IDs have repeated entries from
codex_thread_store::local::thread_history_materialization with the same
invalid type: map, expected f64 error. archived is the current local state;
an archived thread is not necessarily deleted.

01a05aa0-e681-7853-8054-0db7fe82ac89  排查 D2 丢失 session 问题       archived=false
01a05abc-3855-7121-a752-0592f5295edc  查找 Super Agent 架构图         archived=true
01a052ca-ba32-7ba1-9d16-91bd2b970561  sa-oncall                      archived=false
01a052ef-3934-7f51-ba48-fc64749c423c  (no display name)              archived=false
01a057a4-c2dc-79e3-99e1-9910818b57e7  恢复 Super Agent 工作流架构评审  archived=false
01a05969-95d9-7611-a686-0daeb6c97cf3  Lumen BUC d2 synthetic smoke     archived=false
01a05975-a81b-7053-836f-f0838801559d  Lumen BUC d2 smoke retry          archived=false
01a05b7d-032b-71d1-8591-aea70918043b  查找 HAAR 静态 Demo              archived=true
01a05787-1687-73e1-a1d3-3cd6f1471e57  排查状态流图 session 丢失       archived=true
01a05198-b71f-72c0-868e-0c22d0471610  绘制 Super Agent 状态流图       archived=true

Three of the non-archived entries use Codex worktree CWDs rather than the main
repository CWD. That may make them invisible under a project/CWD-filtered
listing even when their thread data is present.

Expected behavior

  • Every valid rollout event should be projected into thread history without halting the projection.
  • A malformed or newly shaped telemetry/token-count field should be ignored, migrated, or recorded as an unknown field rather than stopping the entire thread history projection.
  • A session with archived=false should be discoverable from the correct remote project/working directory and visible in the sidebar/search.
  • Resuming by UUID should show all previously persisted turns, including turns after the first rejected event.

Actual behavior

  • The thread-history projection stops at a fixed ordinal and does not catch up.
  • The sidebar/search can make a live or recently active session appear missing, especially when project/CWD filtering is applied.
  • Resuming by UUID can show only a partial history even though the raw rollout is complete and valid.
  • The error repeats on subsequent projection attempts, so reconnecting alone does not repair the view.

Workarounds currently available

  • Resume by exact UUID, for example:

    codex resume 01a05aa0-e681-7853-8054-0db7fe82ac89
    
  • Use the all-sessions view when the session is hidden by CWD filtering:

    codex resume --all
    
  • Do not delete or rebuild the Codex SQLite databases; doing so could destroy the projection/index state and make recovery harder.

Requested investigation

Please investigate:

  1. Whether a recent Codex rollout/schema change introduced a structured value where the local thread-history materializer still expects an f64.
  2. Whether the projection cursor can recover automatically after a rejected event, while preserving append-only history.
  3. Why session metadata/name/index entries can diverge from state_5.sqlite.threads, causing title/project search to miss an existing session.
  4. Whether the remote Codex app-server and laptop UI can surface projection failures instead of silently presenting an incomplete/missing history.
  5. A safe repair or migration procedure for already-affected threads, without deleting raw rollout JSONL or requiring manual database edits.

Additional diagnostics available on request

I can provide redacted copies of the affected rollout metadata, projection-state rows, and relevant log excerpts. Any diagnostic bundle should be reviewed for repository names, local paths, prompts, and other potentially sensitive content before sharing.

What steps can reproduce the bug?

Uploaded thread: 01a05c51-82df-7291-9070-b1dac92b610b

What is the expected behavior?

No response

Additional information

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with codex_thread_store::local::thread_history_materialization and reproduce the invalid type: map, expected f64 error using the affected rollout JSONL and projection state in thread_history_1.sqlite. Trace how the token_count event advances the cursor and verify that projection continues, preserves later turns, and makes the session discoverable without deleting raw rollout data.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sqlite
Domain
backend, cli, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.