Streaming: tool_use input_json_delta is buffered until complete, causing multi-minute silences on large tool arguments
- Lingua principale
- Shell
- Stelle
- 11.2k
- Fork
- 1.9k
- Merge medio
- 14h 16m
- PR unite (30g)
- 6
Descrizione
## Summary
On `/v1/messages` streaming responses, `input_json_delta` events for a `tool_use` block appear to be buffered until the tool-argument JSON is complete, then flushed all at once. For a large tool argument this produces a multi-minute window in which the connection is open and healthy but carries **zero bytes**.
Text and thinking blocks in the same response stream normally. Only the tool-argument deltas are withheld.
The request ultimately succeeds — the model was working the whole time — but by then any client with an inactivity timeout has already given up. Claude Code aborts at 300s of no content events with `Response stalled mid-stream`, so any single tool call above roughly 31,500 output tokens fails deterministically.
No `ping` events are emitted during the silence.
## Measurements
Same request replayed twice against `/v1/messages`, `claude-opus-5`, streaming, varying only `max_tokens`:
| `max_tokens` | `output_tokens` | longest silence | ms per output token | tool args delivered | `stop_reason` | wall clock |
|---|---|---|---|---|---|---|
| 64000 | 35,899 | **329.5s** | 9.18 | 85,744 chars | `tool_use` | 339.5s |
| 8000 | 8,000 | **78.3s** | 9.79 | **0 chars** | `max_tokens` | 84.4s |
Two independent measurements agree on ms-per-output-token within 6%, so the silence scales linearly with the size of the tool argument being generated:
```
silence ≈ output_tokens × 9.5ms (≈105 tokens/sec)
300s client threshold → ≈31,500 output tokens
```
## Event timeline — 64000 case
```
t= 3.776 message_start
t= 6.488 content_block_delta thinking_delta ← streams normally
t= 6.648 content_block_delta text_delta ← streams normally
t= 6.649 content_block_start tool_use "Write"
t= 6.649 content_block_delta input_json_delta partial_json:"" ← empty
──────────────── 329.5s, zero bytes ────────────────
t=336.140 content_block_delta input_json_delta partial_json:"{\""
t=336.140 …12,215 deltas totalling 85,744 chars…
t=339.410 content_block_stop
t=339.441 message_delta stop_reason: tool_use
t=339.442 message_stop
```
Time to first byte was 3.8s, so the connection itself was fine.
The 85,744 characters arrive in **3.3 seconds** after the silence ends — about 26,000 chars/sec, an order of magnitude faster than the model's own generation rate. That gap is what points at buffering rather than slow generation: the content was produced steadily over 329.5s and released in one burst at the end.
## Secondary issue: truncated tool JSON delivers nothing but bills in full
In the `max_tokens=8000` run the tool-argument JSON was cut off by the token limit. Because the JSON never completed, **not a single character was forwarded**:
```
t= 6.129 content_block_start tool_use "Write"
t= 6.129 content_block_delta input_json_delta partial_json:""
────────── 78.3s silence ──────────
t=84.400 message_delta stop_reason: max_tokens
t=84.400 message_stop
```
The response is `200`, `usage.output_tokens` is 8,000, and the caller receives an empty tool call. All-or-nothing delivery means a caller pays full price for zero usable output whenever a tool argument hits the token limit. Incremental forwarding would at least surface the partial JSON.
## Impact
This is reproducible and not rare — it is a function of output size, not of load or luck. Any workflow that has the model emit one large artifact through a single tool call crosses the threshold.
A concrete example that reproduces every time: the `writing-plans` skill from the Superpowers plugin has the model write a complete implementation plan in a single `Write` call. The resulting document runs to about 2,000 lines, which lands right above the 31,500-token line — so it stalls on essentially every attempt, and the client aborts before the (otherwise successful) response arrives.
The general shape: **one tool call whose argument exceeds ~31,500 output tokens will exceed a 300s client inactivity timeout.**
## What would fix it
Forwarding `input_json_delta` incrementally, the way `text_delta` and `thinking_delta` are already forwarded, would remove the silence entirely — the events exist and are correctly sequenced, they just arrive together at the end.
Failing that, emitting `ping` events during the gap would at least keep inactivity-based clients alive. Currently zero arrive.
## Environment
- Endpoint: `POST /v1/messages` (Anthropic Messages API shape), `stream: true`
- Model: `claude-opus-5`
- `anthropic-beta: claude-code-20250219,context-1m-2025-08-07,context-management-2025-06-27`
- Prompt: 348,483 input tokens, 294 tool definitions
- Observed on 2026-07-29
Happy to supply the full event-by-event timeline for either run.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia dal punto di ingresso dello streaming /v1/messages e riproduci il caso di argomenti dello strumento di grandi dimensioni descritto nell’issue, confrontando la temporizzazione di input_json_delta con quella di text_delta e thinking_delta. Non sono indicati file sorgente o test, quindi segui la gestione dello stream e aggiungi o aggiorna la copertura per la consegna incrementale degli argomenti dello strumento o per il comportamento di keepalive. L’attività è completata quando le chiamate allo strumento grandi e troncate non lasciano più il client silenzioso fino al completamento.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- shell
- Ambito
- api, backend
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100