API 524 on non-streaming requests that take too long (~180s); streaming works fine

Summary

API 524s on non-streaming requests that take too long.

Expected Behavior

API should return the response even if streaming is turned off.

Actual Behavior

API returns 524 on non-streaming requests that take longer than ~180s. Streaming works fine on the same task. The 524 body is the relay's own envelope: {"error":{"type":"server_error","message":"Upstream model provider is temporarily unavailable..."}}. The ceiling appears variable: a non-streaming 16K request succeeded at 109.5s while a 12K request died at 180.2s. This looks like the relay buffering the full non-streaming response and timing out on long generations.

Steps to reproduce the issue

1. Send a non-streaming chat/completions request with high reasoning_effort and a prompt that produces a long generation (>~2 min).
2. The relay holds the connection open while it buffers the response.
3. After ~180s the request dies with HTTP 524.
4. The same request with stream: true succeeds, even past 204s.

Command Code Version

n/a - api

Operating System

Linux

Terminal/IDE

Hermes

Shell

No response

Session file (optional)

No response

Fix prompt (optional)

No response

Additional context

same model (deepseek/deepseek-v4-flash), same prompt, same reasoning_effort=max, direct to api.commandcode.ai/provider/v1:

Non-streaming, max_tokens 12288 → 524 at 180.2s
Non-streaming, max_tokens 16384 → 200 OK at 109.5s
Non-streaming, max_tokens 8192 (hard prompt) → 200 OK at 100.5s
Non-streaming, max_tokens 8192 (easy prompt) → 200 OK at 1.9s
Streaming, max_tokens 16384 (same hard prompt) → 200 OK at 204.6s

commandcode relay A/B test — 2026-08-16 19:00-20:00 PDT

Endpoint: https://api.commandcode.ai/provider/v1/chat/completions

Model: deepseek/deepseek-v4-flash · reasoning_effort=max

Hard prompt: "Think very carefully and at length about the history of computing, step by step, in extreme detail. Cover every era in depth. Write a very long essay."

Easy prompt (control): "Write a haiku about the ocean."

Test scripts: run_tests.py (round 1), run_tests2.py (round 2) — same directory.

{"label": "T8k_control", "status": 200, "elapsed_s": 1.9, "bytes": 988, "ok": true, "finish": "stop", "content_len": 71, "reasoning_len": 0, "usage": {"prompt_tokens": 104, "completion_tokens": 52, "total_tokens": 156, "completion_tokens_details": {"reasoning_tokens": 36}}}
{"label": "T12k_boundary", "status": 524, "elapsed_s": 180.2, "bytes": 127, "ok": false, "api_error": "{'message': 'Upstream model provider is temporarily unavailable. Please try again in a moment.', 'type': 'server_error'}"}
{"label": "T16k_repeat", "status": 200, "elapsed_s": 109.5, "bytes": 68903, "ok": true, "finish": "stop", "content_len": 3491, "reasoning_len": 0, "usage": {"prompt_tokens": 128, "completion_tokens": 8850, "total_tokens": 8978, "completion_tokens_details": {"reasoning_tokens": 8057}}}
{"label": "Tstream_16k", "status": 200, "elapsed_s": 204.6, "bytes": 5421554, "ok": true, "raw_head": "data: {"id":"gen_01M06TNW59PYEZ36TQ5CBYDB5S","object":"chat.completion.chunk","created":1786935702,"model":"deepseek/dee"}
{"label": "T8k_hard", "status": 200, "elapsed_s": 100.5, "bytes": 48345, "ok": true, "finish": "length", "content_len": 28296, "reasoning_len": 0, "usage": {"prompt_tokens": 128, "completion_tokens": 8192, "total_tokens": 8320, "completion_tokens_details": {"reasoning_tokens": 2357}}}

Bắt đầu với endpoint được báo cáo và các script run_tests.py và run_tests2.py được tham chiếu trong cùng thư mục. So sánh các trường hợp không streaming và streaming, sau đó lần theo đường đi của relay chịu trách nhiệm cho các yêu cầu dài. Công việc được xem là hoàn tất khi các lần tạo nội dung dài không streaming trả về thành công thay vì HTTP 524, trong khi hành vi streaming hiện tại vẫn được giữ nguyên.