Streaming: tool_use input_json_delta is buffered until complete, causing multi-minute silences on large tool arguments
- 主要语言
- Shell
- 星标
- 11.2k
- 派生
- 1.9k
- 平均合并
- 14 小时 16 分钟
- 30 天内合并 PR
- 6
描述
## Summary
On `/v1/messages` streaming responses, `input_json_delta` events for a `tool_use` block appear to be buffered until the tool-argument JSON is complete, then flushed all at once. For a large tool argument this produces a multi-minute window in which the connection is open and healthy but carries **zero bytes**.
Text and thinking blocks in the same response stream normally. Only the tool-argument deltas are withheld.
The request ultimately succeeds — the model was working the whole time — but by then any client with an inactivity timeout has already given up. Claude Code aborts at 300s of no content events with `Response stalled mid-stream`, so any single tool call above roughly 31,500 output tokens fails deterministically.
No `ping` events are emitted during the silence.
## Measurements
Same request replayed twice against `/v1/messages`, `claude-opus-5`, streaming, varying only `max_tokens`:
| `max_tokens` | `output_tokens` | longest silence | ms per output token | tool args delivered | `stop_reason` | wall clock |
|---|---|---|---|---|---|---|
| 64000 | 35,899 | **329.5s** | 9.18 | 85,744 chars | `tool_use` | 339.5s |
| 8000 | 8,000 | **78.3s** | 9.79 | **0 chars** | `max_tokens` | 84.4s |
Two independent measurements agree on ms-per-output-token within 6%, so the silence scales linearly with the size of the tool argument being generated:
```
silence ≈ output_tokens × 9.5ms (≈105 tokens/sec)
300s client threshold → ≈31,500 output tokens
```
## Event timeline — 64000 case
```
t= 3.776 message_start
t= 6.488 content_block_delta thinking_delta ← streams normally
t= 6.648 content_block_delta text_delta ← streams normally
t= 6.649 content_block_start tool_use "Write"
t= 6.649 content_block_delta input_json_delta partial_json:"" ← empty
──────────────── 329.5s, zero bytes ────────────────
t=336.140 content_block_delta input_json_delta partial_json:"{\""
t=336.140 …12,215 deltas totalling 85,744 chars…
t=339.410 content_block_stop
t=339.441 message_delta stop_reason: tool_use
t=339.442 message_stop
```
Time to first byte was 3.8s, so the connection itself was fine.
The 85,744 characters arrive in **3.3 seconds** after the silence ends — about 26,000 chars/sec, an order of magnitude faster than the model's own generation rate. That gap is what points at buffering rather than slow generation: the content was produced steadily over 329.5s and released in one burst at the end.
## Secondary issue: truncated tool JSON delivers nothing but bills in full
In the `max_tokens=8000` run the tool-argument JSON was cut off by the token limit. Because the JSON never completed, **not a single character was forwarded**:
```
t= 6.129 content_block_start tool_use "Write"
t= 6.129 content_block_delta input_json_delta partial_json:""
────────── 78.3s silence ──────────
t=84.400 message_delta stop_reason: max_tokens
t=84.400 message_stop
```
The response is `200`, `usage.output_tokens` is 8,000, and the caller receives an empty tool call. All-or-nothing delivery means a caller pays full price for zero usable output whenever a tool argument hits the token limit. Incremental forwarding would at least surface the partial JSON.
## Impact
This is reproducible and not rare — it is a function of output size, not of load or luck. Any workflow that has the model emit one large artifact through a single tool call crosses the threshold.
A concrete example that reproduces every time: the `writing-plans` skill from the Superpowers plugin has the model write a complete implementation plan in a single `Write` call. The resulting document runs to about 2,000 lines, which lands right above the 31,500-token line — so it stalls on essentially every attempt, and the client aborts before the (otherwise successful) response arrives.
The general shape: **one tool call whose argument exceeds ~31,500 output tokens will exceed a 300s client inactivity timeout.**
## What would fix it
Forwarding `input_json_delta` incrementally, the way `text_delta` and `thinking_delta` are already forwarded, would remove the silence entirely — the events exist and are correctly sequenced, they just arrive together at the end.
Failing that, emitting `ping` events during the gap would at least keep inactivity-based clients alive. Currently zero arrive.
## Environment
- Endpoint: `POST /v1/messages` (Anthropic Messages API shape), `stream: true`
- Model: `claude-opus-5`
- `anthropic-beta: claude-code-20250219,context-1m-2025-08-07,context-management-2025-06-27`
- Prompt: 348,483 input tokens, 294 tool definitions
- Observed on 2026-07-29
Happy to supply the full event-by-event timeline for either run.
贡献指南
调研方向
从 /v1/messages 流式入口开始,重现 issue 中描述的大型工具参数案例,并比较 input_json_delta 与 text_delta 和 thinking_delta 的时序。没有指定源文件或测试,因此请跟踪流处理,并为工具参数的增量传递或 keepalive 行为添加或更新覆盖。完成的标志是,大型和被截断的工具调用不再让客户端一直沉默到完成为止。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- shell
- 领域
- api, backend
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 冷清
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100