openai / openai/codex

thread/resume is effectively quadratic on large active threads, blocking Remote steering

Open
#38,787 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

app-server bug performance
Dominant language
Rust
Stars
125k
Forks
19.5k
PR merge metrics
PR metrics pending

Description

Summary

Opening or steering a very large active Codex thread from iOS Remote can time out even though pairing, authentication, transport, and the host app-server are healthy.

The same failure occurs through a client that sends a follow-up to an existing thread: the client first performs thread/resume, app-server parses the rollout successfully, then remains CPU-bound while composing the resume response. The follow-up is never submitted.

For the affected thread, using the already-loaded Codex Desktop composer bypassed thread/resume and delivered the steer immediately.

Environment

  • Codex CLI / app-server: 0.147.0
  • Codex Desktop: 26.810.50856
  • iOS Remote client: 1.2026.209
  • Host: Linux x86_64 (6.17.0-14-generic)
  • Rollout size when reproduced: 22,617 records, dominated by one long active turn with many tool and subagent items

Observed behavior

Each Remote attempt completed rollout parsing quickly:

Resumed rollout with 22617 items, parse errors: 0

It then stopped making progress at:

composing running thread resume response ... active_turn_present=true ... active_turn_status=InProgress

During the stall:

  • app-server consumed approximately one CPU core continuously;
  • RSS was approximately 1.1 GB;
  • the client eventually surfaced a generic failure;
  • the user prompt was absent from the target rollout, confirming that steering never began;
  • repeated iOS retries opened additional connections that all stalled at the same reconstruction point.

There were no concurrent 401, 403, duplicate-owner 409, pairing, or websocket-connectivity failures.

Reproduction

  1. Create or retain an active thread with a very large rollout, particularly one long turn containing thousands of tool/subagent items.
  2. Connect to the host through iOS Remote.
  3. Open that thread or send a prompt while its turn is active.
  4. Observe that the client initiates thread/resume with turn history included.
  5. Rollout parsing completes, but the resume response does not return in practical time and the steer is never submitted.
  6. Open the same already-loaded thread in Codex Desktop and use its composer/Steer control; the message is recorded and handled immediately.

The failure also reproduced through the Codex thread-management follow-up path because it performs the same full thread/resume before steering.

Root cause

In app-server/src/request_processors/thread_lifecycle.rs, the loaded-thread resume path calls populate_thread_turns_from_history whenever pending.include_turns is true:

https://github.com/openai/codex/blob/rust-v0.147.0/codex-rs/app-server/src/request_processors/thread_lifecycle.rs#L528-L567

That builds legacy API turns from every rollout item. During reconstruction, upsert_turn_item searches the growing item vector linearly for every incoming item:

https://github.com/openai/codex/blob/rust-v0.147.0/codex-rs/app-server-protocol/src/protocol/thread_history.rs#L1547-L1559

if let Some(existing_item_index) = items
    .iter()
    .position(|existing_item| existing_item.id() == item.id())

For a turn with thousands of items, repeated linear scans make reconstruction effectively quadratic. The stall begins after parsing and exactly where the logs say app-server is composing the running-thread response. The process behavior is consistent with CPU-bound reconstruction rather than a dead connection or authentication failure.

The same linear upsert was still present on main when checked on 2026-08-15.

Expected behavior

A Remote client should be able to open and steer a known active thread without material latency proportional to the square of its retained turn-item count. A failure should also be bounded and retryable rather than accumulating more expensive resume jobs.

Suggested fixes

Two complementary changes would address this:

  1. Metadata-only steering path: when the client already knows the thread and only needs its active turn ID, call thread/resume with excludeTurns: true, then submit turn/steer with expectedTurnId. Do not serialize full history merely to steer an active thread.
  2. Linear history reconstruction: maintain an item-ID-to-index map while building a turn, or otherwise avoid items.iter().position(...) for every item. Pagination/bounds would provide additional protection for unusually large histories.

It would also help to coalesce or reject duplicate in-flight resumes for the same thread so repeated mobile retries do not amplify the CPU queue.

Workaround

Use the already-loaded Codex Desktop composer and explicitly select Steer. This bypassed full history reconstruction and delivered the prompt immediately. Avoid repeated Remote retries on the affected thread.

Privacy note

This report intentionally omits the thread ID and full rollout because the rollout contains private project conversation and tool output. The two sanitized log lines above identify the failing stage without including that content. I can provide narrower diagnostics if maintainers specify what is needed.

Related but not exact duplicates

  • #37047: thread/resume hangs on a stale active listener with no running turn. Here the turn is genuinely in progress and app-server is CPU-bound reconstructing 22k+ history items.
  • #36831: mobile prompt remains stuck in Status Sending; adjacent symptom, but this report identifies the preceding resume-reconstruction bottleneck.
  • #36416: Remote thread/list scans many rollout files. Similar principle—Remote blocking on unnecessary history work—but a different method and data structure.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in app-server/src/request_processors/thread_lifecycle.rs at the loaded-thread resume path and then inspect app-server-protocol/src/protocol/thread_history.rs, especially populate_thread_turns_from_history and upsert_turn_item. Trace how active-thread resume reconstructs history, then validate that a large active rollout can resume and be steered without quadratic reconstruction or unbounded retry work.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.