`response.incomplete` is treated as a retryable stream failure, causing unnecessary retries and transport fallback

What version of Codex CLI is running?

Official rust-v0.147.0 at be6e8eac029b183056b7e4402879f15d2c85f61b. I reproduced the bug in a clean detached worktree of that tag with a deterministic codex-api mock-SSE test. The same affected handling is present in current main at 73abda8bfef6bd42eb11351be53980a027fd1feb.

What subscription do you have?

Pro

Which model were you using?

gpt-5.6-sol. The handling bug is model-independent.

What platform is your computer?

Microsoft Windows NT 10.0.19045.0 x64

What terminal emulator and version are you using (if applicable)?

Windows Terminal 1.24.11911.0 with PowerShell 7.6.4. The behavior is in the Responses event handling path and is terminal-independent.

Codex doctor report

Not included because this has a deterministic mock-SSE reproduction and does not depend on installation, authentication, terminal, MCP, or network configuration.

What issue are you seeing?

When the provider cleanly ends a Responses stream with a response.incomplete event, Codex converts it to ApiError::Stream. For example:

{
  "type": "response.incomplete",
  "response": {
    "status": "incomplete",
    "incomplete_details": {
      "reason": "max_output_tokens"
    },
    "usage": {
      "input_tokens": 120,
      "output_tokens": 30,
      "total_tokens": 150
    }
  }
}

This is a semantic terminal event from the provider, not a dropped network stream. However, mapping it to ApiError::Stream makes the session loop treat it as retryable. After the retry budget is exhausted, the same classification can also trigger the WebSocket-to-HTTPS fallback:

Falling back from WebSockets to HTTPS transport. stream disconnected before completion: Incomplete response returned, reason: max_output_tokens

stream disconnected before completion: Incomplete response returned, reason: max_output_tokens

Changing transports cannot fix max_output_tokens or content_filter, and repeating the same request can add latency and usage. The parser also discards the usage block carried by response.incomplete, leaving session token accounting stale.

What steps can reproduce the bug?

1. Configure a mock Responses endpoint to return an SSE response.incomplete event with incomplete_details.reason = "max_output_tokens" and a usage block.
2. Set stream_max_retries to a value greater than zero.
3. Start a turn against the mock endpoint.
4. Observe that the event becomes a retryable stream error rather than a terminal incomplete response. The retry path is entered and the supplied usage is not recorded.

The same behavior follows directly from these current paths:

• codex-rs/codex-api/src/sse/responses.rs: response.incomplete is converted to ApiError::Stream.
• codex-rs/codex-api/src/api_bridge.rs: ApiError::Stream becomes CodexErr::Stream.
• codex-rs/protocol/src/error.rs: stream errors are retryable.
• codex-rs/core/src/responses_retry.rs: retry exhaustion can switch from WebSockets to HTTPS.

The existing incomplete_response_emits_content_filter_error_message integration test sets stream_max_retries = 0, so it verifies the displayed error but cannot detect that the event is incorrectly classified as retryable. Setting retries to 2 and asserting that the mock receives exactly one request exposes the classification bug.

What is the expected behavior?

response.incomplete should remain distinct from transport failure. Codex should:

• preserve incomplete_details.reason;
• preserve and record the event's usage block when present;
• surface a clear terminal error for reasons such as max_output_tokens and content_filter;
• avoid stream retries and transport fallback for this semantic terminal event.

Automatic continuation is a separate policy question because blindly replaying a partially completed response can duplicate tool side effects.

Additional information

The focused repair is to add a dedicated ResponseIncomplete API/protocol error carrying the reason and optional token usage, classify it as non-retryable, and update session accounting before returning it. If usage is absent, Codex can recompute the local context estimate rather than leaving the last count unchanged.

Candidate implementation:

• Branch: https://github.com/starriet9/codex/tree/fix/response-incomplete-terminal
• Commit: https://github.com/starriet9/codex/commit/e556049f66f09a318fcf73a92979941b334e5e29

I implemented and tested this candidate repair locally against main. The candidate branch is based on current main at 73abda8bfef6bd42eb11351be53980a027fd1feb; none of the affected files changed between the tested revision and that commit:

• a characterization test in a clean official rust-v0.147.0 worktree confirmed that a response.incomplete event carrying usage becomes a retryable ApiError::Stream;
• 168 codex-api tests passed;
• 273 codex-protocol tests passed;
• 3 codex-response-debug-context tests passed;
• focused integration tests verify that content_filter and max_output_tokens are terminal and non-retryable and each makes exactly one request;
• the max_output_tokens integration test verifies that usage is preserved, and both cases produce a clear reason-specific error;
• Clippy passed for all changed packages.

Related but distinct:

• #14753 reports the max_output_tokens symptom but was closed without a root-cause fix.
• #37138 covers missing token accounting when usage is absent and notes that response.incomplete usage is discarded, but it does not cover the retry and transport-fallback misclassification.
• #11558 introduced the generic response.incomplete handling that currently maps the event to a stream error.

Start in codex-rs/codex-api/src/sse/responses.rs, then trace ApiError::Stream through api_bridge.rs, protocol/src/error.rs, and core/src/responses_retry.rs. Run the existing incomplete-response integration test and extend it with retries enabled and a request count. Done means response.incomplete remains terminal, preserves its reason and usage, and does not retry or trigger transport fallback.