openai / openai/codex

Rollout writer reuses the previous ordinal when `thread_settings_applied` follows a non-durable `token_count`, permanently stalling that thread's durable history projection

Open
#44,609 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

app bug session
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

Summary

When a thread_settings_applied event is appended to a rollout immediately after a
non-durable token_count event, the writer assigns it the same ordinal as the
token_count
. The rollout file then contains a duplicate ordinal, and the durable
history projector — which enforces strict monotonic ordinals — rejects it with:

failed to project durable rollout for <thread_id>: thread-store internal error:
thread history projection for <thread_id> expected ordinal 492, got 491

The projection cursor never advances past that point again. Every subsequent append
re-throws the same error for the life of the thread (392 occurrences on one of mine).
That thread's durable history is frozen from that moment on, silently.

User-visible symptom

The live event stream is unaffected, so nothing looks wrong while you stay in the
conversation — new turns stream in normally. The damage only shows when something
re-reads stored history:

  • Reopening the thread shows it truncated at the stall point.
  • On the mobile app it is worse, because it has no live buffer: the conversation loads
    only up to the stall, you can reply and watch that reply stream fine, and then it
    vanishes again the moment you leave and come back.

There is no error surfaced in the UI. The messages look lost.

Environment

  • Codex app 26.903.9818.0 (MSIX, OpenAI.Codex_2p2nqsd0c76g0)
  • codex-cli 0.153.4
  • ChatGPT Desktop 1.2026.190.0
  • Windows 11 10.0.26200
  • Also reproduced on threads created under 0.150.0-alpha.12.2 and 0.152.1, so this
    is not new to 0.153.4.

Evidence

Every corrupted rollout has exactly one duplicated ordinal, and in all five cases the
duplicate is a token_count followed by a thread_settings_applied reusing its ordinal:

line 492: {"timestamp":"2026-09-10T14:26:16.022Z","ordinal":491,"type":"event_msg",
           "payload":{"type":"token_count", ...
line 493: {"timestamp":"2026-09-10T14:26:56.774Z","ordinal":491,"type":"event_msg",
           "payload":{"type":"thread_settings_applied", ...

The delay between the two varies from 30 seconds to 8 hours, so this is a settings-apply
landing on an idle thread whose last recorded event was a token_count.

Controlled comparison inside a single rollout

That same file contains nine thread_settings_applied events. Eight follow a durable
event and receive a correct fresh ordinal. The one that follows a token_count
duplicates. Nothing else differs:

line ordinal preceding event result
366 365 task_complete (364) ok
382 381 turn_aborted (380) ok
493 491 token_count (491) duplicate
534 532 response item (531) ok
595 593 task_complete (592) ok
618 616 task_complete (615) ok
723 721 task_complete (720) ok
1017 1015 response item (1014) ok
1302 1300 task_complete (1299) ok

This points at the next-ordinal value being derived from the last durable item while
token_count still consumes an ordinal in the rollout file — the same non-durable
token_count ordinal accounting already described in #40747.

Failing call path
persist_rollout_items{item_count=1}:append_items{item_count=1}:append_items{item_count=1}:
  failed to project durable rollout for <thread_id>: thread-store internal error:
  thread history projection for <thread_id> expected ordinal 492, got 491

Target: codex_thread_store::local::live_writer. The same error also fires on the
shutdown path (op.dispatch.shutdown → "failed to project durable rollout during
shutdown"), so a clean exit does not repair it either.

Blast radius

5 of 800 local threads over roughly twelve days, earliest 2026-08-30. Combined, about
52 MB / ~1,300 rollout lines of history became unreadable while appearing intact in the
live view. Two of the five were hit 85 ms apart, which suggests an app-level settings
apply fanning out across open threads rather than any per-thread user action.

Useful discriminator when triaging: a thread whose projection cursor is behind its
rollout file but has no duplicate ordinal is just normal live lag and catches up. The
duplicate ordinal is what makes it permanent.

Detection

For each row in thread_history_projection_state, compare next_rollout_byte_offset to
the real size of the matching sessions/**/rollout-*.jsonl. Any thread that is behind
and whose file contains a duplicate "ordinal":N is permanently stalled.

Local repair (works, in case it helps others)

The stored byte offset already points at the duplicate line; only the ordinal is one too
high. With the app fully exited and the DB backed up:

UPDATE thread_history_projection_state
SET next_rollout_ordinal = next_rollout_ordinal - 1
WHERE thread_id IN (...);

On reopening the thread, the projector consumed the remaining 27.9 MB and caught up
completely — cursor 491 → 1346, durable items 171 → 505, zero bytes behind, no further
errors. Recovered items included the user messages that had gone missing, so nothing is
actually lost on disk; it is purely a projection-cursor problem.

Note the projector is lazy — it backfills on thread open or next append, not at startup.

Suggested directions

  1. Assign the rollout ordinal from the actual last written rollout ordinal rather than
    the last durable item, so non-durable events like token_count cannot leave a gap
    that a later append reuses.
  2. Have the projector treat expected N+1, got N on a known non-durable predecessor as
    recoverable — skip or resync the cursor — instead of hard-failing forever.
  3. Surface a projection stall in the UI. Silent, permanent truncation of stored history
    that still looks healthy in the live view is the worst part of this; it is only
    visible in logs, and only if you know the string to grep for.

Related

  • #40747 — resume fails when the inherited-prefix ordinal points at a non-durable
    token_count; same accounting area.
  • #33241 — divergent rollout histories from two writers.
  • #16599 — lost thread history after app restart; possibly the same root cause seen from
    the user side.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the codex_thread_store::local::live_writer call path and trace how rollout ordinals are assigned after non-durable token_count events. Inspect the thread history projection state handling and reproduce the token_count followed by thread_settings_applied sequence. Done means the resulting rollout has strictly advancing ordinals and durable projection no longer stalls or repeatedly reports the ordinal mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.