thread/resume is effectively quadratic on large active threads, blocking Remote steering
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 125k
- Forks
- 19.5k
- PR merge metrics
- PR metrics pending
Description
Summary
Opening or steering a very large active Codex thread from iOS Remote can time out even though pairing, authentication, transport, and the host app-server are healthy.
The same failure occurs through a client that sends a follow-up to an existing thread: the client first performs thread/resume, app-server parses the rollout successfully, then remains CPU-bound while composing the resume response. The follow-up is never submitted.
For the affected thread, using the already-loaded Codex Desktop composer bypassed thread/resume and delivered the steer immediately.
Environment
- Codex CLI / app-server:
0.147.0 - Codex Desktop:
26.810.50856 - iOS Remote client:
1.2026.209 - Host: Linux x86_64 (
6.17.0-14-generic) - Rollout size when reproduced: 22,617 records, dominated by one long active turn with many tool and subagent items
Observed behavior
Each Remote attempt completed rollout parsing quickly:
Resumed rollout with 22617 items, parse errors: 0
It then stopped making progress at:
composing running thread resume response ... active_turn_present=true ... active_turn_status=InProgress
During the stall:
- app-server consumed approximately one CPU core continuously;
- RSS was approximately 1.1 GB;
- the client eventually surfaced a generic failure;
- the user prompt was absent from the target rollout, confirming that steering never began;
- repeated iOS retries opened additional connections that all stalled at the same reconstruction point.
There were no concurrent 401, 403, duplicate-owner 409, pairing, or websocket-connectivity failures.
Reproduction
- Create or retain an active thread with a very large rollout, particularly one long turn containing thousands of tool/subagent items.
- Connect to the host through iOS Remote.
- Open that thread or send a prompt while its turn is active.
- Observe that the client initiates
thread/resumewith turn history included. - Rollout parsing completes, but the resume response does not return in practical time and the steer is never submitted.
- Open the same already-loaded thread in Codex Desktop and use its composer/Steer control; the message is recorded and handled immediately.
The failure also reproduced through the Codex thread-management follow-up path because it performs the same full thread/resume before steering.
Root cause
In app-server/src/request_processors/thread_lifecycle.rs, the loaded-thread resume path calls populate_thread_turns_from_history whenever pending.include_turns is true:
That builds legacy API turns from every rollout item. During reconstruction, upsert_turn_item searches the growing item vector linearly for every incoming item:
if let Some(existing_item_index) = items
.iter()
.position(|existing_item| existing_item.id() == item.id())
For a turn with thousands of items, repeated linear scans make reconstruction effectively quadratic. The stall begins after parsing and exactly where the logs say app-server is composing the running-thread response. The process behavior is consistent with CPU-bound reconstruction rather than a dead connection or authentication failure.
The same linear upsert was still present on main when checked on 2026-08-15.
Expected behavior
A Remote client should be able to open and steer a known active thread without material latency proportional to the square of its retained turn-item count. A failure should also be bounded and retryable rather than accumulating more expensive resume jobs.
Suggested fixes
Two complementary changes would address this:
- Metadata-only steering path: when the client already knows the thread and only needs its active turn ID, call
thread/resumewithexcludeTurns: true, then submitturn/steerwithexpectedTurnId. Do not serialize full history merely to steer an active thread. - Linear history reconstruction: maintain an item-ID-to-index map while building a turn, or otherwise avoid
items.iter().position(...)for every item. Pagination/bounds would provide additional protection for unusually large histories.
It would also help to coalesce or reject duplicate in-flight resumes for the same thread so repeated mobile retries do not amplify the CPU queue.
Workaround
Use the already-loaded Codex Desktop composer and explicitly select Steer. This bypassed full history reconstruction and delivered the prompt immediately. Avoid repeated Remote retries on the affected thread.
Privacy note
This report intentionally omits the thread ID and full rollout because the rollout contains private project conversation and tool output. The two sanitized log lines above identify the failing stage without including that content. I can provide narrower diagnostics if maintainers specify what is needed.
Related but not exact duplicates
- #37047:
thread/resumehangs on a stale active listener with no running turn. Here the turn is genuinely in progress and app-server is CPU-bound reconstructing 22k+ history items. - #36831: mobile prompt remains stuck in
Status Sending; adjacent symptom, but this report identifies the preceding resume-reconstruction bottleneck. - #36416: Remote
thread/listscans many rollout files. Similar principle—Remote blocking on unnecessary history work—but a different method and data structure.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in app-server/src/request_processors/thread_lifecycle.rs at the loaded-thread resume path and then inspect app-server-protocol/src/protocol/thread_history.rs, especially populate_thread_turns_from_history and upsert_turn_item. Trace how active-thread resume reconstructs history, then validate that a large active rollout can resume and be steered without quadratic reconstruction or unbounded retry work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100