Mid-stream Chutes transport failures are unrecoverable and often untyped (error decoding response body)

Start with crates/inference_providers/src/attested/chutes/mod.rs:1516, then trace sse_error_frame() in crates/api/src/routes/completions.rs and retry_with_fallback_caps in crates/services/src/inference_provider_pool/mod.rs. Compare the four proposed handling options against the three observed failure shapes and existing pre-stream retry behavior. Done requires a decided, testable approach for mid-stream transport failures rather than an unresolved design discussion.

Summary

When the Chutes upstream byte stream breaks after cloud-api has returned HTTP 200 and started forwarding chunks, the client gets a dead partial stream with nothing actionable, and nothing retries or recovers. This is the one remaining unfixed item from the BeeZi feedback report (2026-07-08, #private-inference): their captured error chunk

{"error":{"message":"Failed to perform completion: Chutes stream transport: error decoding response body","type":"server_error"}}

Where it comes from

• crates/inference_providers/src/attested/chutes/mod.rs:1516 (line as of 24eeb176) wraps reqwest byte-stream errors as CompletionError::CompletionError("Chutes stream transport: {e}"); "error decoding response body" is reqwest's Display for a body cut mid-read.
• sse_error_frame() in crates/api/src/routes/completions.rs serializes it with type server_error, then the stream ends with [DONE].
• retry_with_fallback_caps in crates/services/src/inference_provider_pool/mod.rs wraps only stream establishment. classify_retry_decision even marks decoding response body as a retryable connection keyword — but that classification never runs once chunks have been emitted, because the 200 + partial output are already committed to the client.
• The June–July Chutes PRs (#798, #810, #860, #866, #870) are all pre-stream fallback/retry/observability or chunk cosmetics; #866 made pre-stream transport failures retryable 503s and explicitly left the mid-stream wrap untouched.

Evidence it fires in prod today (post prod-20260716-24eeb17)

• Live repro 2026-07-20: 3 of 15 streaming attempts broke mid-stream on deepseek/deepseek-v3.2 (0/3 on z-ai/glm-5, 0/3 on moonshotai/kimi-k2.5, 0/36 on minimax/minimax-m2.5 incl. 24 at 4-way concurrency — so currently model-concentrated, not path-wide).
• The three observed failure shapes, in order of decreasing client ergonomics:
  1. structured in-stream server_error chunk followed by [DONE] (the BeeZi capture);
  2. abrupt TCP/chunked-encoding cut — no error chunk, no [DONE];
  3. silent stall until the client's own read timeout (120s+ of nothing).
• Datadog (us3): 10 Completion stream error events with error_type=completion_error on deepseek/deepseek-v3.2 on 2026-07-18/19, i.e. after the current release was deployed.

Options to discuss

1. Retry before first forwarded chunk: if the upstream stream dies after establishment but before any content chunk has been forwarded to the client, the request is still transparently retryable (same semantics as the existing pre-stream retry). Probably the highest-value/lowest-risk slice, but requires moving the retry boundary out of the pool or making the forwarding layer retry-aware.
2. Typed + actionable terminal error frame: distinguish upstream_transport_error from generic server_error, and include a retryable: true hint, so agent platforms can implement client-side resume-by-retry.
3. Truncation marker on shape (2)/(3): when the upstream disconnects cleanly without finish_reason, emit an explicit error/incomplete frame instead of appending a bare [DONE] (today silent truncation is indistinguishable from a normal end of stream except by the missing finish_reason).
4. Full mid-stream resume (re-prompt with partial output) — probably not worth the complexity.

Raw repro data and per-attempt logs are available from the 2026-07-20 triage session if useful.