Streaming path cannot detect or escape a stalled backend (dsv4 incident 2026-07-20/21)

Context: DeepSeek-V4-Flash streaming incident — engine-side decode droughts froze all in-flight streams simultaneously for 20-30s+ (see Notion "deepseek-v4-streaming-incident" and cvm-compose-files#145 for the engine-side mitigation). The gateway amplified the damage: every layer below held the dead streams open and kept routing to the affected backends, with zero error logs anywhere. All items verified against origin/main on 2026-07-21:

1. No inter-chunk idle timeout on the NEAR path. crates/inference_providers/src/attested/nearai/mod.rs sets read_timeout from the completion timeout (~600s), so a stalled stream hangs the client for up to 10 minutes. The Chutes client already does this right: read_timeout(90) in crates/inference_providers/src/attested/chutes/client.rs. Match it (with care for legitimately slow long-context decodes — consider ~90-120s inter-chunk).

2. Backend health scoring is TTFT-only. crates/inference_providers/src/attested/nearai/fleet.rs record_ttft is the only writer to backend_stats — a backend that returns its first token fast and then stalls scores as healthy and keeps receiving traffic. Add an inter-token-latency and/or mid-stream-failure signal.

3. Retries never cover mid-flight failures. crates/services/src/inference_provider_pool/mod.rs — the retry closure returns once response headers arrive; any later failure is terminal for the client.

4. sglang delivers queue-full 503s on streaming requests as an in-band SSE error under an already-committed HTTP 200 (upstream fix sgl-project/sglang#28175 is unmerged), so the 503-keyed retry_with_fallback never fires for streaming. Since --max-queued-requests was added to dsv4 (2026-07-06), every streaming queue-full has surfaced to customers as a dead 200 stream. Detect the in-band error frame and fail over / surface a real error.

5. Clean upstream EOF is minted into a synthetic data: [DONE] (route tail in crates/api/src/routes/completions.rs), making a mid-generation backend disconnect indistinguishable from a complete response. The E2EE path already guards this (e2ee_stream.rs); the plaintext path should error on EOF without an inner [DONE].

Items 1-3 and 5 were independently identified in Henry's incident writeup; this issue tracks them for implementation.

Start with the streaming timeout and health-scoring paths in crates/inference_providers/src/attested/nearai/mod.rs and fleet.rs, comparing the Chutes client’s read_timeout(90). Then trace retries in crates/services/src/inference_provider_pool/mod.rs and EOF handling in crates/api/src/routes/completions.rs, including the E2EE path. Done means stalled or queue-full streams fail visibly, unhealthy backends are avoided, and clean completion remains distinguishable from mid-stream disconnects.