API 524 on non-streaming requests that take too long (~180s); streaming works fine

Summary

API 524s on non-streaming requests that take too long.

Expected Behavior

API should return the response even if streaming is turned off.

Actual Behavior

API returns 524 on non-streaming requests that take longer than ~180s. Streaming works fine on the same task. The 524 body is the relay's own envelope: {"error":{"type":"server_error","message":"Upstream model provider is temporarily unavailable..."}}. The ceiling appears variable: a non-streaming 16K request succeeded at 109.5s while a 12K request died at 180.2s. This looks like the relay buffering the full non-streaming response and timing out on long generations.

Steps to reproduce the issue

1. Send a non-streaming chat/completions request with high reasoning_effort and a prompt that produces a long generation (>~2 min).
2. The relay holds the connection open while it buffers the response.
3. After ~180s the request dies with HTTP 524.
4. The same request with stream: true succeeds, even past 204s.

Command Code Version

n/a - api

Operating System

Linux

Terminal/IDE

Hermes

Shell

No response

Session file (optional)

No response

Fix prompt (optional)

No response

Additional context

same model (deepseek/deepseek-v4-flash), same prompt, same reasoning_effort=max, direct to api.commandcode.ai/provider/v1:

Non-streaming, max_tokens 12288 → 524 at 180.2s
Non-streaming, max_tokens 16384 → 200 OK at 109.5s
Non-streaming, max_tokens 8192 (hard prompt) → 200 OK at 100.5s
Non-streaming, max_tokens 8192 (easy prompt) → 200 OK at 1.9s
Streaming, max_tokens 16384 (same hard prompt) → 200 OK at 204.6s

commandcode relay A/B test — 2026-08-16 19:00-20:00 PDT

Endpoint: https://api.commandcode.ai/provider/v1/chat/completions

Model: deepseek/deepseek-v4-flash · reasoning_effort=max

Hard prompt: "Think very carefully and at length about the history of computing, step by step, in extreme detail. Cover every era in depth. Write a very long essay."

Easy prompt (control): "Write a haiku about the ocean."

Test scripts: run_tests.py (round 1), run_tests2.py (round 2) — same directory.

{"label": "T8k_control", "status": 200, "elapsed_s": 1.9, "bytes": 988, "ok": true, "finish": "stop", "content_len": 71, "reasoning_len": 0, "usage": {"prompt_tokens": 104, "completion_tokens": 52, "total_tokens": 156, "completion_tokens_details": {"reasoning_tokens": 36}}}
{"label": "T12k_boundary", "status": 524, "elapsed_s": 180.2, "bytes": 127, "ok": false, "api_error": "{'message': 'Upstream model provider is temporarily unavailable. Please try again in a moment.', 'type': 'server_error'}"}
{"label": "T16k_repeat", "status": 200, "elapsed_s": 109.5, "bytes": 68903, "ok": true, "finish": "stop", "content_len": 3491, "reasoning_len": 0, "usage": {"prompt_tokens": 128, "completion_tokens": 8850, "total_tokens": 8978, "completion_tokens_details": {"reasoning_tokens": 8057}}}
{"label": "Tstream_16k", "status": 200, "elapsed_s": 204.6, "bytes": 5421554, "ok": true, "raw_head": "data: {"id":"gen_01M06TNW59PYEZ36TQ5CBYDB5S","object":"chat.completion.chunk","created":1786935702,"model":"deepseek/dee"}
{"label": "T8k_hard", "status": 200, "elapsed_s": 100.5, "bytes": 48345, "ok": true, "finish": "length", "content_len": 28296, "reasoning_len": 0, "usage": {"prompt_tokens": 128, "completion_tokens": 8192, "total_tokens": 8320, "completion_tokens_details": {"reasoning_tokens": 2357}}}

Comienza con el endpoint reportado y los scripts referenciados run_tests.py y run_tests2.py en el mismo directorio. Compara los casos sin streaming y con streaming, y después sigue el recorrido del relay responsable de las solicitudes largas. Se considera terminado cuando las generaciones largas sin streaming se devuelven correctamente en lugar de HTTP 524, mientras el comportamiento de streaming existente permanece intacto.