Responses WebSocket black-hole after VPN disconnect leaves CLI Thinking for 5 minutes
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 125k
- Forks
- 19.4k
- PR merge metrics
- PR metrics pending
Description
Environment
| Field | Value |
|---|---|
| Codex CLI | 0.153.4 |
| Authentication | ChatGPT |
| Subscription | ChatGPT account; tier not recorded |
| Model | gpt-5.6-terra |
| Platform | Linux 7.1.8-arch1-2-ptl x86_64 unknown |
| Terminal | Ghostty 1.3.1-arch2 |
What issue are you seeing?
The original symptom was disconnecting a VPN while an active Codex CLI turn was still shown as Thinking. The CLI then appeared to think indefinitely (at least several minutes) instead of detecting the loss and reconnecting. Killing Codex and starting it again after connectivity was restored worked immediately.
I reproduced the underlying failure with a controlled network black hole. A local CONNECT proxy allowed the authenticated Responses WebSocket to establish, then stopped forwarding bytes in both directions while deliberately retaining both TCP connections. This simulates a VPN/path-routing loss where no FIN or RST reaches the client.
With the normal default configuration, Codex remained in the active turn for 299.987 seconds after traffic stopped. At 15 seconds into the stall, ss still reported the Codex-to-proxy and proxy-to-upstream connections as ESTABLISHED with empty queues. The first error was:
stream disconnected before completion: idle timeout waiting for websocket
The first retry began immediately at that point, but the release UI did not display it. The replacement WebSocket handshake then consumed its 15-second connection timeout, and only retry two showed:
Reconnecting... 2/5 (request timed out)
Starting a new direct Codex turn after stopping the black-hole proxy completed in about five seconds. This explains why restart appears to fix the original VPN scenario: it abandons the wedged socket rather than waiting for its passive read timeout.
Steps to reproduce the bug
- Start an authenticated Codex CLI turn through the Responses WebSocket path and wait until it is active/Thinking.
- While keeping the existing TCP connection open, remove the route that carries its traffic. A VPN disconnect is one real-world trigger. For a deterministic test, use a CONNECT proxy that forwards the initial TLS/WebSocket traffic, then drops all subsequent bytes in both directions without closing either socket.
- Observe the active turn and the client socket.
Controlled timeline (UTC):
| Event | Timestamp |
|---|---|
| Request successfully sent | 02:30:12.098Z |
| Proxy starts black-holing traffic | 02:30:29.793Z |
ss snapshot still shows ESTAB |
02:30:44Z |
| First stream error | 02:35:29.780Z |
| Retry one / replacement connect begins | 02:35:29.780Z / 02:35:30.003Z |
| First visible reconnect, retry two | 02:35:45.005Z |
As a control, configuring a custom provider with stream_idle_timeout_ms = 12000 changed detection from 299.987 seconds to 11.990 seconds. This confirms the default stream idle timeout is what gates detection. Lowering it globally is not a safe fix because healthy reasoning can produce no response frames for that length of time.
Expected behavior
Codex should detect that an established Responses WebSocket is no longer live promptly, show a reconnecting status at the first retry, and preserve legitimate long-running reasoning that has no user-visible output.
Investigation
codex-rs/model-provider-info/src/lib.rssetsDEFAULT_STREAM_IDLE_TIMEOUT_MS = 300_000.run_websocket_response_streamincodex-rs/codex-api/src/endpoint/responses_websocket.rspassively wrapsws_stream.next()in that timeout.- The Responses WebSocket pump replies to a server Ping with Pong, but it does not originate client Ping frames or require a Pong deadline.
- In this failure, the timeout is
ApiError::Stream("idle timeout waiting for websocket"), mapped byapi_bridge.rstoCodexErr::Stream. - The unbounded retry branch in
core/src/responses_retry.rsapplies only toCodexErrorDetails::ConnectionFailed, so this takes the ordinary bounded stream-retry path. The unbounded-retry feature is not the source of the five-minute detection delay. - Retry one is intentionally hidden in the release UI; this makes the stall look worse but is secondary to transport liveness detection.
codex-rs/app-server-transport/src/transport/remote_control/websocket.rs already has the relevant pattern: periodic client Ping plus a bounded Pong deadline.
Proposed direction and tested prototype
I have a small local prototype against responses_websocket.rs (166 additions, 4 deletions). It:
- sends a WebSocket Ping every 15 seconds;
- fails the connection if no Pong arrives within 45 seconds;
- retains the independent 300-second response-data idle timeout, so Pongs prove transport liveness but do not make a silent model response run forever.
Two local raw-WebSocket tests passed:
mise exec rust@1.95.0 -- just test -p codex-api websocket_liveness_watchdog
# 2 passed, 182 skipped
They verify (a) an unanswered Ping returns websocket pong timeout, and (b) a silent peer that replies to Ping reaches the existing response-data idle timeout instead of being killed by the watchdog. I can turn this prototype into a PR if maintainers agree with the approach.
Diagnostics
codex doctor --json was healthy before reproduction. Relevant redacted results:
{
"overallStatus": "ok",
"codexVersion": "0.153.4",
"auth": "ChatGPT configured",
"model": "gpt-5.6-terra",
"provider": "openai",
"network.websocket_reachability": "Responses WebSocket handshake succeeded",
"websocket_connect_timeout_ms": 15000,
"wire_api": "responses"
}
Raw traces are available locally but include authentication-related HTTP headers, so I have not attached them publicly. Safe timestamps and socket-state evidence are summarized above.
Related: #20739 (WebSocket closes before response.completed) and #24260 (long visible stall). This is distinct from #18471, which appears to be a Desktop conversation-state race.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with run_websocket_response_stream in codex-rs/codex-api/src/endpoint/responses_websocket.rs and compare its behavior with codex-rs/app-server-transport/src/transport/remote_control/websocket.rs. Run the websocket_liveness_watchdog tests with the documented mise command, then inspect the retry handling in core/src/responses_retry.rs and api_bridge.rs. Done means stalled connections are detected promptly, legitimate silent responses still reach the existing data-idle timeout, and the relevant reconnect status is visible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100