migrate-rollouts silently drops >16 MiB JSONL records while reporting migration successful

Start in codex-rs/thread-store/src/local/rollout_migration.rs and read the migration tests in codex-rs/thread-store/src/local/rollout_migration_tests.rs. Reproduce oversized custom_tool_call_output and compacted records, including a later valid event, then compare dry-run and apply behavior. Done means oversized valid records are not silently lost and tests cover reporting, source replacement safety, and readback fidelity.

What version of Codex CLI is running?

0.148.0-alpha.5

What subscription do you have?

Not applicable (local rollout migration).

Which model were you using?

Not applicable (local rollout migration).

What platform is your computer?

macOS / Darwin 27.0.0 / arm64

What terminal are you using?

zsh

Codex doctor report

Not included: this is a deterministic local storage transformation and source-level reproduction with no auth, network, or model dependency.

What is the issue?

codex migrate-rollouts --apply can silently omit valid legacy JSONL records larger than 16 MiB while reporting the thread as successfully migrated.

The current implementation sets:

const MAX_ROLLOUT_LINE_BYTES: usize = 16 * 1024 * 1024;

When a record is larger than that limit, read_rollout_record() consumes and discards the whole record through its newline and returns no line to the canonicalizer. The record is therefore absent from the staged paginated rollout and is not recoverable from the SQLite projection. Migration can still finish with status = migrated and no warning or message.

The size check happens before JSON parsing and is not restricted to tool output records, so it also affects valid compacted checkpoints.

Dry-run does not detect this. For a legacy rollout it reads session metadata and returns eligible without scanning all records, so the first full-record scan and oversized-record handling occur only during --apply.

The existing migration test also demonstrates the behavior: a valid oversized function_call_output is absent after migration while the outcome remains migrated.

Steps to reproduce

1. Create a valid legacy rollout JSONL containing normal session metadata and normal events.
2. Add a syntactically valid JSONL record larger than 16 MiB, for example:
   • a response_item / custom_tool_call_output containing inline image data; or
   • a compacted record whose replacement_history exceeds 16 MiB.
3. Add another valid event after the oversized record to show that processing continues.
4. Run a dry-run:

   codex migrate-rollouts --thread <THREAD_UUID> --json
   

   The rollout is reported as eligible; the oversized record is not reported.
5. Back up the test rollout, then run:

   codex migrate-rollouts --apply --thread <THREAD_UUID> --json
   

6. Compare the migrated rollout with the input.

What did you expect to happen?

Migration should never silently turn a valid rollout into a partial rollout.

Preferably, migration should preserve valid records losslessly. If a finite per-record limit is required, then:

• dry-run should scan for and report oversized records;
• apply should fail closed before replacing the source, or require an explicit opt-in to omit data;
• any partial result should not use the unqualified migrated status;
• output should report skipped record counts, sizes, and positions;
• a recoverable backup should exist before source replacement.

What happened instead?

• The oversized record is consumed through its newline but not passed to canonicalization.
• Migration continues with later records.
• The staged file replaces the original rollout without the oversized record.
• The outcome can be migrated with message = null.
• There is no skipped-record count, byte count, ordinal, sentinel, or warning.
• Surviving records are assigned canonical ordinals, so there is no ordinal gap to reveal the loss.
• Dry-run reports eligible because it does not perform the full scan.

For a dropped custom_tool_call_output, resume may only synthesize an aborted output if the corresponding call remains; the actual output is not restored.

For a dropped compacted { replacement_history } record, resume may replay surviving raw events, but that is not necessarily equivalent to the compacted model-visible history and its checkpoint/baseline state. A newer surviving checkpoint can reduce runtime impact, but it does not restore archival fidelity.

Additional information

I performed a read-only, content-safe metadata audit of three existing legacy rollouts. I did not run --apply on them.

Across those three files, there were:

• 10 syntactically valid records larger than 16 MiB;
• 242,416,165 total bytes in those records (about 231 MiB);
• 7 response_item / custom_tool_call_output records dominated by inline image data;
• 3 compacted / replacement_history records dominated by image-bearing history.

This shows the limit is reachable in real Codex sessions, not only with artificial terminal output. No session payloads, inline images, thread IDs, workspace paths, or credentials are included in this report.

Relevant implementation and tests:

• CLI migration/reporting: https://github.com/openai/codex/pull/37348
• Oversized-record handling: https://github.com/openai/codex/pull/37191
• Current implementation: https://github.com/openai/codex/blob/main/codex-rs/thread-store/src/local/rollout_migration.rs
• Current tests: https://github.com/openai/codex/blob/main/codex-rs/thread-store/src/local/rollout_migration_tests.rs

Related but distinct reports:

• https://github.com/openai/codex/issues/37670 — migration fidelity issue involving duplicated compatibility input, not silent oversized-record loss
• https://github.com/openai/codex/issues/28531 — image-heavy inline rollout data causing Desktop crashes/freezes
• https://github.com/openai/codex/issues/33735 — compacted.replacement_history duplicating inline images and growing rollout size

Suggested regression coverage:

• 16 MiB custom_tool_call_output;

• 16 MiB compacted / replacement_history;

• a valid record after the oversized record;
• dry-run/apply consistency;
• resume/readback fidelity;
• explicit non-success status whenever any record is omitted.