openai / openai/codex

Severe CLI reliability failure: 63.8 MB image-history requests before any compaction, WebSocket fallback, and prolonged stalls on Windows

Open
#43,015 11 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug CLI connectivity context performance windows-os
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

Summary and urgent request

Please urgently investigate aggregate image-history byte growth and the repeated full-request cost after WebSocket fallback, and provide a supported recovery path for affected sessions.

In a real image-assisted coding task, Codex accumulated 63,759,837 bytes per request, still 45,853,691 bytes after zstd compression. At the corresponding history snapshot, 38 typed image occurrences accounted for 61,279,872 ASCII data-URL characters. This happened before any compaction, not only through images retained in an old compacted replacement history.

The failure had a serious practical impact: repeated stream errors, five-attempt retry batches, HTTPS fallback, and prolonged periods in which the primary task did not advance. One failed sequence on 0.153.2 lasted approximately 88 minutes without new effective model output. Updating to 0.153.4, the latest stable release when checked on September 5, did not remove the accumulated payload or prevent the observed WebSocket fallback.

I am extremely dissatisfied with this experience. These failure periods made Codex CLI effectively unusable for my paid development workflow. Waiting through prolonged retries without an actionable explanation or reliable recovery procedure is unacceptable. Users should not need to inspect SQLite databases, count Base64 payloads, or study client source code merely to keep an image-assisted coding task usable. I am asking for urgent maintainer triage, a supported mitigation, and a prioritized fix—not merely another generic reconnect warning.

The report also includes a later partial recovery: a successful compaction removed the old tool-image payloads from active replacement history and work resumed. I am not claiming permanent data loss or that every use of view_image fails.

Related reports and the additional evidence here

This is closely related to #41338 and #24550. Please consolidate this evidence into an existing tracker if that is the preferred triage path.

The distinguishing measurements are:

  • Windows, the built-in OpenAI provider, ChatGPT authentication, and current stable 0.153.4—not a Responses-to-Chat bridge.
  • Zero compaction records at the failure snapshot: ordinary current-window image outputs can already create the large-request condition.
  • 37 distinct data URLs across 38 occurrences. Exact deduplication would remove only 1,119,846 bytes, or 1.83% of the image data.
  • The largest individual data URL was 5,183,170 bytes. None exceeded 10 MiB. A per-image 10 MiB ceiling alone would not address this workload.
  • Actual pre/post-compression request measurements, separately from rollout file size and token accounting.
  • A subsequent compaction that can be inspected structurally: tool-output images were removed, while the original user-message images remained.

Environment

Field Observed value
Codex CLI 0.153.2 during the original long failed sequence; upgraded to 0.153.4 before the later resumed failure
Current stable release checked rust-v0.153.4, published September 4, 2026
Source version audited 3d2ee51ca2d5db578f328aa75e20aa22c0197c9a / rust-v0.153.4
Platform Windows x64, OS build 10.0.26100
Shell PowerShell 7.6; terminal-emulator version not collected
Subscription / authentication ChatGPT Pro; ChatGPT authentication
Provider Built-in OpenAI provider, Codex Responses endpoint
Model gpt-6-astra
Reasoning selection Initially xhigh; the resumed pre-compaction and post-compaction turns both used ultra
Service tier default
Network Explicit HTTP/HTTPS proxy variables pointing to a local v2rayN proxy; TUN/routing changes are disclosed below

Relevant non-default context settings were:

model_context_window = 1000000
model_auto_compact_token_limit = 900000

The session reported a model_context_window of 828,400. I am disclosing the configured values without asserting that they were the effective runtime limits. This is not a claim of reproduction with an untouched default configuration. It is a supported long-context workflow that lacked an effective safeguard against its much larger serialized image payload.

Codex doctor report

A new full codex doctor --json report was not collected for this investigation. The evidence below comes from targeted read-only inspection of the affected rollout, retained client logs, process/network state, and the pinned client source. I am not substituting an older doctor report or claiming that doctor passed.

What happened: measured request and history sizes

The HTTP client recorded the following at 2026-09-05 11:12:14 UTC:

Compressed request body with zstd
pre_compression_bytes=63759837
post_compression_bytes=45853691
compression_duration_ms=78

Several earlier requests in the same resumed task were almost as large:

UTC Before compression After compression Compression time
10:55:20 63,721,050 B 45,827,415 B 70 ms
10:57:42 63,740,637 B 45,841,192 B 76 ms
10:58:22 63,742,371 B 45,842,100 B 70 ms
10:59:43 63,742,371 B 45,842,094 B 98 ms
11:12:14 63,759,837 B 45,853,691 B 78 ms

These are application-recorded request-body sizes, not inferred wire traffic, billing totals, or the size of the JSONL file. Compression itself was fast; the measurements do not justify blaming local zstd work for multi-minute delays.

The corresponding history was inspected through 11:12:15 UTC, counting only response_item records:

Typed image location Occurrences Data-URL characters / ASCII bytes
User-message content[].image_url 7 4,307,974
Function/custom-tool output[].image_url 31 56,971,898
Total 38 61,279,872

All counted image values belonged to typed input_image objects. UI/event duplicates were not included in these totals. The main sources were generated images and image-viewing tool outputs; this is not a claim that the whole total came from view_image alone.

The last completed request recorded at 10:58:22 UTC reported 482,479 input tokens, of which 480,000 were cached. That is the last completed-request usage record, not a new usage measurement for the 11:12 submission. It demonstrates that high token-cache reuse coexisted with repeated large requests; unchanged usage counters do not prove the backend performed no work or incurred no cost.

Failure chronology and severity

All times below are September 5, 2026, UTC.

Time / interval Observation
08:53:15–10:21:18, CLI 0.153.2 After the last productive tool result, repeated response-stream failures produced no new effective model output for approximately 88 minutes; the turn eventually failed.
09:08:19, 09:23:24, 09:38:29, 09:51:07, 10:06:11 Five recorded retries in that failed sequence. The roughly 15-minute spacing is observed behavior, not a claim about a proven timeout setting.
10:51–10:52 CLI upgraded to 0.153.4 and restarted; the original session was resumed.
10:53 Multiple WebSocket handshakes succeeded; actual request sends then failed with Windows error 10054 and exhausted five retries.
10:54:01 HTTP fallback was activated.
10:55–10:58 Some productive tool activity resumed, including starting two subagents.
10:59:43 A separate manual route change restarted the proxy and caused a known disconnect. This event is treated separately, as explained below.
11:00:19 and 11:12:44 HTTPS requests received HTTP 200; SSE keepalives were also observed during long waits.
11:07:35 and 11:08:44 The two subagents completed; their progress must not be confused with the primary thread's progress.
11:31:23–11:36:38 The running turn was interrupted, then a compaction ran and completed.

The errors included:

stream disconnected before completion:
failed to send websocket request: IO error: ... (os error 10054)

Falling back from WebSockets to HTTPS transport

stream disconnected before completion:
Transport error: network error: error decoding response body

The original 88-minute terminal failure occurred on 0.153.2. The later handshake/send failures, fallback and large requests were observed on 0.153.4. I am not attributing the earlier 88-minute sequence to the newer binary.

Later recovery: important evidence, not omitted

At 11:36:38, the completed compaction installed a replacement history containing:

  • 36 items: 35 messages and one compaction item;
  • 7 embedded images totaling 4,307,974 characters;
  • approximately 4,378,186 bytes when serialized as compact UTF-8 JSON by the inspection code.

The retained image multiset exactly matched the earlier user-message image multiset. The old raw tool outputs were no longer present in the replacement history. Relative to the earlier snapshot, embedded image bytes decreased by approximately 92.97%.

Subsequent retained HTTP logs showed requests around 9.8–10.9 MB before compression, substantially below the earlier 63.8 MB, and the primary thread again performed commands and file edits. Those later requests already included additional post-compaction work, so they are not the size of the first untouched compacted request.

This is a natural before/after observation, not a controlled benchmark. The model selection remained Astra/Ultra and the same CLI process continued, but the workload changed and upstream conditions were not controlled. Long waits were not eliminated: one later parent-thread tool-result-to-next-tool-call interval was still about 11 minutes.

The observed recovery supports treating active-history size as an important contributor. It does not prove that compaction alone fixes every affected thread, or that every minute of delay was upload time. It also matches an important implementation distinction: current compaction can discard original tool outputs while retaining user images.

Reproduction conditions

The observed workload was:

  1. Run a long image-assisted coding session using the built-in OpenAI provider and the environment described above.
  2. Generate and inspect multiple images through normal tools. Retain several dozen prepared images in active history; individual images need not exceed 10 MiB.
  3. Continue model/tool iterations without a compaction having reduced that history.
  4. Resume the same history after a CLI restart.
  5. Compare request-body byte measurements with token accounting, and observe whether the actual resumed WebSocket request succeeds after the handshake.

This describes the real failure conditions. A standalone minimal deterministic network reproducer and a sanitized-copy A/B test have not been run. The private workload cannot be pasted into a public issue. The report is not evidence that every image, network, model, or default configuration reproduces the transport failure.

Technical assessment: what is proven, and what is most likely

Confirmed: this workload retained tens of megabytes of typed image data and repeatedly constructed very large requests. The current source explains how that payload survives truncation and is resent when the full-history path is used.

Most strongly supported explanation of the amplification: accumulated inline images become expensive repeatedly when incremental WebSocket continuation is unavailable and the client submits the full active prompt over HTTP. Additional images can enlarge subsequent submissions again. The natural compaction observation and related community controls support this explanation.

Not established: the exact origin of the first TCP reset, a particular server/proxy message-size ceiling, or a universal causal relationship between image size and every observed stall. Payload size or multimodal replay shape may trigger a transport/endpoint limitation, but local/upstream proxy behavior and backend processing are still possible contributors. High reasoning effort also contributes to ordinary model latency.

Pinned source evidence and the conditional byte-amplification argument

The following code was checked at release commit 3d2ee51ca2d5db578f328aa75e20aa22c0197c9a:

Component Relevant behavior
Image-viewing handler Produces a data URL and a typed image output. Omitting the image from the log preview does not remove it from model input.
Tool-output truncation Text is budgeted/truncated while InputImage URLs are copied in the inspected paths.
History/token estimation Replaces the actual Base64 byte contribution with an image-token estimate. Correct token estimation still requires a separate serialized-byte safeguard.
Request construction and transports HTTP constructs a request from the formatted active prompt. WS may use incremental input when prior-response state and prefix matching permit it; otherwise it uses full input.
Compacted-history retention Filters ordinary raw tool outputs; retained user images can remain. Thus compaction outcomes depend on the history's shape.
WebSocket response stream The ordinary Message::Close(_) branch discards close details and returns a generic stream error. This does not imply that our 10054 included a close frame.

The deterministic part can be stated narrowly. Let b_i be the byte length of each prepared inline image URL retained in a history that is serialized in full. These strings must contribute at least sum(b_i) bytes to that uncompressed request, before surrounding JSON and other inputs. If the same image strings are retained across k such requests, they contribute at least k * sum(b_i) bytes across the constructed request bodies.

That is a serialization consequence under stated conditions. It is not proof that all requests reach the network in full, that those bytes are billable tokens, or that a connection must fail at a particular size. Compression and WS incremental continuation change transfer costs; the recorded zstd sizes establish the substantial cost in this case.

For this workload, a single-image ceiling and exact deduplication are insufficient on their own: no image exceeded 10 MiB, and 37 of 38 encoded images were distinct. The safeguard needs an aggregate/lifecycle dimension.

Confounders and limits explicitly checked

  • Proxy route change: at 10:59:43, changing routing restarted Xray/sing-box and disrupted both the affected session and another active session. That disconnect is not used as proof of image-induced failure. Earlier failures predated it.
  • Proxy configuration was present: explicit HTTP/HTTPS proxy settings were active and actual WebSocket handshakes succeeded. This differs from an older missing-proxy/handshake-timeout problem.
  • Connection versus progress: HTTP 200 and periodic SSE keepalives demonstrate some established stream activity, not completion of model work. I am not calling every long wait a deadlock.
  • Local resource evidence: the inspected snapshots did not show CPU/memory exhaustion; local request compression was measured in milliseconds. These snapshots do not exclude every possible local bottleneck.
  • Billing: no monetary loss, duplicate model billing, or precise quota waste is claimed from the byte counts or unchanged usage counters.
  • Recovery: a compaction later succeeded and work resumed. No permanent data loss or permanently unrecoverable session is claimed.
  • Log retention: older retained-client-log rows rotated during the investigation. Their measurements were captured while present; the persistent rollout and saved diagnostic observations were cross-checked. No unrestricted packet capture or server-side trace was obtained.

Existing fixes and reports have been considered

Reference Relevance and boundary
#41338 Directly reports the byte/token mismatch from local image viewing; open when checked.
#24550 Reports WS retries/fallback with retained images. Its original compaction-specific reproduction differs from this pre-compaction case.
#30441 Related image-history size failures; this local report does not claim a 413 was observed.
#24388 and closure response Acknowledges an earlier compaction issue and explains a change in compaction implementation. This should not be read as confirmation that every image-byte retention problem is fixed.
Merged PR #38681 Preserves parent HTTP fallback for delegated sessions. Its commit is an ancestor of 0.153.4 and the relevant code is present. I am not claiming that this specific fix is absent.

In particular, I am not requesting that maintainers redo #38681. Please assess the remaining current-window byte growth and repeated full-history submission separately from already-addressed delegated-session fallback behavior.

Expected behavior and urgent requested action

Image viewing is a core, legitimate workflow. The expected solution should preserve useful visual capability rather than requiring users to avoid images altogether.

Please:

  1. Urgently triage and assign ownership for the aggregate image-history/request-byte problem, clarify which existing tracker owns it, and provide a current supported mitigation and expected fix/release path.
  2. Add an aggregate serialized-byte safeguard, independent of token budgeting, covering ordinary tool outputs before compaction as well as continuation, resume/fork and compacted history. Many individually small images must be covered.
  3. Provide a bounded lifecycle or out-of-band representation for older image artifacts, with explicit references and on-demand retrieval. Do not silently discard task-critical current images or the user's original files.
  4. Avoid repeatedly resubmitting a deterministically rejected large request unchanged. Preserve actionable close/status details when available; do not infer an oversize error solely from a generic TCP reset.
  5. Provide a supported recovery procedure for existing affected sessions, with preservation of project files and useful task context, instead of requiring manual JSONL/SQLite surgery or abandoning a long task without a reliable handoff.
  6. Make the UI distinguish connection attempts, accepted requests with heartbeats, actual model progress, and repeated request-size failure. Surface the relevant byte measurements and a useful recovery action.

Suggested regression coverage should include multiple sub-10-MiB images, current-window tool images before any compaction, high token-cache reuse with large serialized bodies, both user and tool images after compaction, and repeated continuation after fallback. Please verify both the request bytes and successful follow-up behavior; a low token count or a successful compaction notification alone is insufficient.

This is a serious interruption to paid, real development work, not a cosmetic warning. Please provide an actionable official response as soon as possible.

Privacy and supporting material

Only sanitized measurements and code references are included. Raw rollouts, images, prompts, project paths, credentials, cookies, account identifiers and private request IDs are intentionally omitted. Please specify an appropriate secure channel if additional private diagnostic material is necessary for triage.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the pinned Codex source version, retained rollout history, and HTTP client logs described in the report. Compare active-history image payload sizes with WebSocket failures and HTTPS fallback, and inspect the compaction before/after evidence. Done means an agreed supported mitigation or recovery path addresses repeated large requests without relying on manual database inspection.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.