Rollout writer reuses the previous ordinal when `thread_settings_applied` follows a non-durable `token_count`, permanently stalling that thread's durable history projection
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 125k
- Forks
- 19.4k
- PR merge metrics
- PR metrics pending
Description
Summary
When a thread_settings_applied event is appended to a rollout immediately after a
non-durable token_count event, the writer assigns it the same ordinal as the
token_count. The rollout file then contains a duplicate ordinal, and the durable
history projector — which enforces strict monotonic ordinals — rejects it with:
failed to project durable rollout for <thread_id>: thread-store internal error:
thread history projection for <thread_id> expected ordinal 492, got 491
The projection cursor never advances past that point again. Every subsequent append
re-throws the same error for the life of the thread (392 occurrences on one of mine).
That thread's durable history is frozen from that moment on, silently.
User-visible symptom
The live event stream is unaffected, so nothing looks wrong while you stay in the
conversation — new turns stream in normally. The damage only shows when something
re-reads stored history:
- Reopening the thread shows it truncated at the stall point.
- On the mobile app it is worse, because it has no live buffer: the conversation loads
only up to the stall, you can reply and watch that reply stream fine, and then it
vanishes again the moment you leave and come back.
There is no error surfaced in the UI. The messages look lost.
Environment
- Codex app
26.903.9818.0(MSIX,OpenAI.Codex_2p2nqsd0c76g0) codex-cli 0.153.4- ChatGPT Desktop
1.2026.190.0 - Windows 11
10.0.26200 - Also reproduced on threads created under
0.150.0-alpha.12.2and0.152.1, so this
is not new to0.153.4.
Evidence
Every corrupted rollout has exactly one duplicated ordinal, and in all five cases the
duplicate is a token_count followed by a thread_settings_applied reusing its ordinal:
line 492: {"timestamp":"2026-09-10T14:26:16.022Z","ordinal":491,"type":"event_msg",
"payload":{"type":"token_count", ...
line 493: {"timestamp":"2026-09-10T14:26:56.774Z","ordinal":491,"type":"event_msg",
"payload":{"type":"thread_settings_applied", ...
The delay between the two varies from 30 seconds to 8 hours, so this is a settings-apply
landing on an idle thread whose last recorded event was a token_count.
Controlled comparison inside a single rollout
That same file contains nine thread_settings_applied events. Eight follow a durable
event and receive a correct fresh ordinal. The one that follows a token_count
duplicates. Nothing else differs:
| line | ordinal | preceding event | result |
|---|---|---|---|
| 366 | 365 | task_complete (364) |
ok |
| 382 | 381 | turn_aborted (380) |
ok |
| 493 | 491 | token_count (491) |
duplicate |
| 534 | 532 | response item (531) | ok |
| 595 | 593 | task_complete (592) |
ok |
| 618 | 616 | task_complete (615) |
ok |
| 723 | 721 | task_complete (720) |
ok |
| 1017 | 1015 | response item (1014) | ok |
| 1302 | 1300 | task_complete (1299) |
ok |
This points at the next-ordinal value being derived from the last durable item while
token_count still consumes an ordinal in the rollout file — the same non-durable
token_count ordinal accounting already described in #40747.
Failing call path
persist_rollout_items{item_count=1}:append_items{item_count=1}:append_items{item_count=1}:
failed to project durable rollout for <thread_id>: thread-store internal error:
thread history projection for <thread_id> expected ordinal 492, got 491
Target: codex_thread_store::local::live_writer. The same error also fires on the
shutdown path (op.dispatch.shutdown → "failed to project durable rollout during
shutdown"), so a clean exit does not repair it either.
Blast radius
5 of 800 local threads over roughly twelve days, earliest 2026-08-30. Combined, about
52 MB / ~1,300 rollout lines of history became unreadable while appearing intact in the
live view. Two of the five were hit 85 ms apart, which suggests an app-level settings
apply fanning out across open threads rather than any per-thread user action.
Useful discriminator when triaging: a thread whose projection cursor is behind its
rollout file but has no duplicate ordinal is just normal live lag and catches up. The
duplicate ordinal is what makes it permanent.
Detection
For each row in thread_history_projection_state, compare next_rollout_byte_offset to
the real size of the matching sessions/**/rollout-*.jsonl. Any thread that is behind
and whose file contains a duplicate "ordinal":N is permanently stalled.
Local repair (works, in case it helps others)
The stored byte offset already points at the duplicate line; only the ordinal is one too
high. With the app fully exited and the DB backed up:
UPDATE thread_history_projection_state
SET next_rollout_ordinal = next_rollout_ordinal - 1
WHERE thread_id IN (...);
On reopening the thread, the projector consumed the remaining 27.9 MB and caught up
completely — cursor 491 → 1346, durable items 171 → 505, zero bytes behind, no further
errors. Recovered items included the user messages that had gone missing, so nothing is
actually lost on disk; it is purely a projection-cursor problem.
Note the projector is lazy — it backfills on thread open or next append, not at startup.
Suggested directions
- Assign the rollout ordinal from the actual last written rollout ordinal rather than
the last durable item, so non-durable events liketoken_countcannot leave a gap
that a later append reuses. - Have the projector treat
expected N+1, got Non a known non-durable predecessor as
recoverable — skip or resync the cursor — instead of hard-failing forever. - Surface a projection stall in the UI. Silent, permanent truncation of stored history
that still looks healthy in the live view is the worst part of this; it is only
visible in logs, and only if you know the string to grep for.
Related
- #40747 — resume fails when the inherited-prefix ordinal points at a non-durable
token_count; same accounting area. - #33241 — divergent rollout histories from two writers.
- #16599 — lost thread history after app restart; possibly the same root cause seen from
the user side.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the codex_thread_store::local::live_writer call path and trace how rollout ordinals are assigned after non-durable token_count events. Inspect the thread history projection state handling and reproduce the token_count followed by thread_settings_applied sequence. Done means the resulting rollout has strictly advancing ordinals and durable projection no longer stalls or repeatedly reports the ordinal mismatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100