Bug: "Reconnecting... waiting for network" is shown for local-IO/stream stalls (not only real network) — common for heavy long-session users, hours wasted per occurrence

Reconnecting... waiting for network is shown for local-IO / stream-stall failures, and the underlying hazard is preventable

This is not a nitpick about wording: the message actively falsifies the diagnosis, and even a correctly-labeled variant would still hit the exact workflow codex is built for — long-lived sessions, several open at once. The end goal should be that users don't experience this error at all, in two layers: (1) when it does fire, report the real cause; (2) address the local-IO hazard so it stops firing in the first place.

codex-cli 0.151.0 shows Reconnecting... waiting for network / Connection failed: error sending request on any streaming Responses request that fails with a reqwest send error, regardless of actual cause. What we hit: a session whose rollout jsonl had grown to a few hundred MB, resumed while several other codex sessions were open at once. Appending to that file while sibling sessions contend for the shared logs_2.sqlite / thread_history_1.sqlite WALs can fire the send/stream deadline — and codex reports that as "waiting for network." Network was fine throughout: curl to the endpoint's /v1/models and /v1/responses (streaming SSE) returned 200 and streamed normally, the API key was valid, codex doctor was all green, and single-session resume made it work again instantly. So the real, recurring hazard for heavy users is local: a giant append-only rollout write path that stalls the stream — not the network.

The mislabeling itself comes from two spots:

• codex-rs/core/src/responses_retry.rs:74 — on CodexErrorDetails::ConnectionFailed it calls sess.notify_stream_error(turn_context, "Reconnecting... waiting for network", err) with a hardcoded string and no look at the underlying reqwest error (DNS / TCP reset / send deadline / body-write stall / resource exhaustion are all named "network").
• codex-rs/protocol/src/error.rs:504 + codex-rs/http-client/src/error.rs:21 — ConnectionFailedError prints Connection failed: {source}, where source is reqwest's generic SendRequest display ("error sending request for url"), which covers timeouts and stalls too.

Two asks, the second being what actually removes the user-facing pain:

1. Report the real cause. Stop saying "waiting for network" for every ConnectionFailed; bucket by reqwest error kind — DNS / connection refused → "cannot reach the model endpoint"; send/stream deadline / body stall → "stream request timed out," with a hint that this can be local load on a very large session (closing other sessions and codex exec resume <id> recovers from the sqlite projection, which already holds the full history). Keep the raw error in tracing::warn!. This is the surgical fix.

2. Stop the hazard from firing. The root cause is that session logging (rollout jsonl append + the SQLite projections) shares the hot path with streaming: slow local IO can stall the response stream into a deadline. If the write path is decoupled from the response path — e.g. async/background/batched flush of session events so a blocked disk never blocks the SSE body pump — the expensive-local-IO scenario stops producing these errors at all. That is what would actually keep heavy multi-session users on codex unbothered; the mislabeling fix alone still leaves them hitting correctly-labeled timeouts.

Patch for the surgical part is committed (eafd4ee) and mirrored on a personal fork branch fix/reconnect-error-not-network, ready to cherry-pick: "Reconnecting... waiting for network" → "Reconnecting... model stream request failed", a comment explaining the why, and the matching test assertion. No retry/recovery/session logic touched. Happy to adjust phrasing, split the classification change into its own PR, or help draft the async-flush design if any of that helps the second ask land.

cd codex-rs && RUST_MIN_STACK=33554432 cargo test -p codex-core --test all stream_no_completed
# → 2 passed; 0 failed
# macOS debug-profile default thread stack is too small → stack overflow; 32 MB via RUST_MIN_STACK fixes it (pre-existing, unrelated to this patch).

diff --git a/codex-rs/core/src/responses_retry.rs b/codex-rs/core/src/responses_retry.rs
index fb05054..26d23da 100644
--- a/codex-rs/core/src/responses_retry.rs
+++ b/codex-rs/core/src/responses_retry.rs
@@ -71,7 +71,14 @@ pub(crate) async fn handle_retryable_response_stream_error(
             ?retry_delay,
             "stream connection failed; waiting to retry"
         );
-        sess.notify_stream_error(turn_context, "Reconnecting... waiting for network", err)
+        // Surface a neutral retry message. We deliberately do NOT say "waiting for
+        // network": a reqwest SendRequest failure is not solely a network outage — under
+        // heavy local load (e.g. a very large session rollout jsonl being appended, or
+        // sibling sessions contending for the shared logs_2 / thread_history SQLite WALs)
+        // the send/stream can hit a timeout/stall that surfaces as the same reqwest error,
+        // misdirecting users into debugging proxies/WiFi/DNS/API keys for hours. The raw
+        // `err` is kept via the `warn!` above and is carried in CodexErr for diagnostics.
+        sess.notify_stream_error(turn_context, "Reconnecting... model stream request failed", err)
             .await;
         retry_state.connection_retries = retry_state.connection_retries.saturating_add(1);
         codex_client::record_retry!(retry_state.connection_retries, retry_delay, operation);
diff --git a/codex-rs/core/tests/suite/stream_no_completed.rs b/codex-rs/core/tests/suite/stream_no_completed.rs
index d9741d0..f38d5b9 100644
--- a/codex-rs/core/tests/suite/stream_no_completed.rs
+++ b/codex-rs/core/tests/suite/stream_no_completed.rs
@@ -133,7 +133,7 @@ async fn connection_failure_pauses_retry_budget_until_provider_is_reachable() ->
     };
     assert_eq!(
         connection_error.message,
-        "Reconnecting... waiting for network"
+        "Reconnecting... model stream request failed"
     );
 
     let recovered_server = MockServer::builder()

Start with codex-rs/core/src/responses_retry.rs:74, then inspect codex-rs/protocol/src/error.rs:504 and codex-rs/http-client/src/error.rs:21 to trace the reported failure. Run codex-rs/core/tests/suite/stream_no_completed.rs with the stated RUST_MIN_STACK setting. Done means the user-facing message no longer attributes every ConnectionFailed error to the network, with the broader local-IO hazard treated separately.