Streaming: tool_use input_json_delta is buffered until complete, causing multi-minute silences on large tool arguments

Summary

On /v1/messages streaming responses, input_json_delta events for a tool_use block appear to be buffered until the tool-argument JSON is complete, then flushed all at once. For a large tool argument this produces a multi-minute window in which the connection is open and healthy but carries zero bytes.

Text and thinking blocks in the same response stream normally. Only the tool-argument deltas are withheld.

The request ultimately succeeds — the model was working the whole time — but by then any client with an inactivity timeout has already given up. Claude Code aborts at 300s of no content events with Response stalled mid-stream, so any single tool call above roughly 31,500 output tokens fails deterministically.

No ping events are emitted during the silence.

Measurements

Same request replayed twice against /v1/messages, claude-opus-5, streaming, varying only max_tokens:

max_tokens	output_tokens	longest silence	ms per output token	tool args delivered	stop_reason	wall clock
64000	35,899	329.5s	9.18	85,744 chars	tool_use	339.5s
8000	8,000	78.3s	9.79	0 chars	max_tokens	84.4s

Two independent measurements agree on ms-per-output-token within 6%, so the silence scales linearly with the size of the tool argument being generated:

silence ≈ output_tokens × 9.5ms      (≈105 tokens/sec)
300s client threshold  →  ≈31,500 output tokens

Event timeline — 64000 case

t=  3.776  message_start
t=  6.488  content_block_delta   thinking_delta        ← streams normally
t=  6.648  content_block_delta   text_delta            ← streams normally
t=  6.649  content_block_start   tool_use "Write"
t=  6.649  content_block_delta   input_json_delta   partial_json:""     ← empty
           ──────────────── 329.5s, zero bytes ────────────────
t=336.140  content_block_delta   input_json_delta   partial_json:"{\""
t=336.140  …12,215 deltas totalling 85,744 chars…
t=339.410  content_block_stop
t=339.441  message_delta         stop_reason: tool_use
t=339.442  message_stop

Time to first byte was 3.8s, so the connection itself was fine.

The 85,744 characters arrive in 3.3 seconds after the silence ends — about 26,000 chars/sec, an order of magnitude faster than the model's own generation rate. That gap is what points at buffering rather than slow generation: the content was produced steadily over 329.5s and released in one burst at the end.

Secondary issue: truncated tool JSON delivers nothing but bills in full

In the max_tokens=8000 run the tool-argument JSON was cut off by the token limit. Because the JSON never completed, not a single character was forwarded:

t= 6.129  content_block_start  tool_use "Write"
t= 6.129  content_block_delta  input_json_delta  partial_json:""
          ────────── 78.3s silence ──────────
t=84.400  message_delta        stop_reason: max_tokens
t=84.400  message_stop

The response is 200, usage.output_tokens is 8,000, and the caller receives an empty tool call. All-or-nothing delivery means a caller pays full price for zero usable output whenever a tool argument hits the token limit. Incremental forwarding would at least surface the partial JSON.

Impact

This is reproducible and not rare — it is a function of output size, not of load or luck. Any workflow that has the model emit one large artifact through a single tool call crosses the threshold.

A concrete example that reproduces every time: the writing-plans skill from the Superpowers plugin has the model write a complete implementation plan in a single Write call. The resulting document runs to about 2,000 lines, which lands right above the 31,500-token line — so it stalls on essentially every attempt, and the client aborts before the (otherwise successful) response arrives.

The general shape: one tool call whose argument exceeds ~31,500 output tokens will exceed a 300s client inactivity timeout.

What would fix it

Forwarding input_json_delta incrementally, the way text_delta and thinking_delta are already forwarded, would remove the silence entirely — the events exist and are correctly sequenced, they just arrive together at the end.

Failing that, emitting ping events during the gap would at least keep inactivity-based clients alive. Currently zero arrive.

Environment

• Endpoint: POST /v1/messages (Anthropic Messages API shape), stream: true
• Model: claude-opus-5
• anthropic-beta: claude-code-20250219,context-1m-2025-08-07,context-management-2025-06-27
• Prompt: 348,483 input tokens, 294 tool definitions
• Observed on 2026-07-29

Happy to supply the full event-by-event timeline for either run.

Bắt đầu tại điểm vào streaming /v1/messages và tái hiện trường hợp đối số công cụ lớn được mô tả trong issue, so sánh thời điểm của input_json_delta với text_delta và thinking_delta. Không có tệp nguồn hoặc test nào được nêu tên, vì vậy hãy lần theo phần xử lý stream và thêm hoặc cập nhật coverage cho việc phân phối tăng dần các đối số công cụ hoặc hành vi keepalive. Được xem là hoàn tất khi các lệnh gọi công cụ lớn và bị cắt ngắn không còn khiến client im lặng cho đến khi hoàn tất.