github / github/copilot-cli

Streaming: tool_use input_json_delta is buffered until complete, causing multi-minute silences on large tool arguments

Open
#4,286 0 comments 0 reactions 0 assignees View on GitHub
area:models area:networking
Dominant language
Shell
Stars
11.2k
Forks
1.9k
Avg merge
14h 16m
Merged PRs (30d)
6

Description

## Summary

On `/v1/messages` streaming responses, `input_json_delta` events for a `tool_use` block appear to be buffered until the tool-argument JSON is complete, then flushed all at once. For a large tool argument this produces a multi-minute window in which the connection is open and healthy but carries **zero bytes**.

Text and thinking blocks in the same response stream normally. Only the tool-argument deltas are withheld.

The request ultimately succeeds — the model was working the whole time — but by then any client with an inactivity timeout has already given up. Claude Code aborts at 300s of no content events with `Response stalled mid-stream`, so any single tool call above roughly 31,500 output tokens fails deterministically.

No `ping` events are emitted during the silence.

## Measurements

Same request replayed twice against `/v1/messages`, `claude-opus-5`, streaming, varying only `max_tokens`:

| `max_tokens` | `output_tokens` | longest silence | ms per output token | tool args delivered | `stop_reason` | wall clock |
|---|---|---|---|---|---|---|
| 64000 | 35,899 | **329.5s** | 9.18 | 85,744 chars | `tool_use` | 339.5s |
| 8000 | 8,000 | **78.3s** | 9.79 | **0 chars** | `max_tokens` | 84.4s |

Two independent measurements agree on ms-per-output-token within 6%, so the silence scales linearly with the size of the tool argument being generated:

```
silence ≈ output_tokens × 9.5ms (≈105 tokens/sec)
300s client threshold → ≈31,500 output tokens
```

## Event timeline — 64000 case

```
t= 3.776 message_start
t= 6.488 content_block_delta thinking_delta ← streams normally
t= 6.648 content_block_delta text_delta ← streams normally
t= 6.649 content_block_start tool_use "Write"
t= 6.649 content_block_delta input_json_delta partial_json:"" ← empty
──────────────── 329.5s, zero bytes ────────────────
t=336.140 content_block_delta input_json_delta partial_json:"{\""
t=336.140 …12,215 deltas totalling 85,744 chars…
t=339.410 content_block_stop
t=339.441 message_delta stop_reason: tool_use
t=339.442 message_stop
```

Time to first byte was 3.8s, so the connection itself was fine.

The 85,744 characters arrive in **3.3 seconds** after the silence ends — about 26,000 chars/sec, an order of magnitude faster than the model's own generation rate. That gap is what points at buffering rather than slow generation: the content was produced steadily over 329.5s and released in one burst at the end.

## Secondary issue: truncated tool JSON delivers nothing but bills in full

In the `max_tokens=8000` run the tool-argument JSON was cut off by the token limit. Because the JSON never completed, **not a single character was forwarded**:

```
t= 6.129 content_block_start tool_use "Write"
t= 6.129 content_block_delta input_json_delta partial_json:""
────────── 78.3s silence ──────────
t=84.400 message_delta stop_reason: max_tokens
t=84.400 message_stop
```

The response is `200`, `usage.output_tokens` is 8,000, and the caller receives an empty tool call. All-or-nothing delivery means a caller pays full price for zero usable output whenever a tool argument hits the token limit. Incremental forwarding would at least surface the partial JSON.

## Impact

This is reproducible and not rare — it is a function of output size, not of load or luck. Any workflow that has the model emit one large artifact through a single tool call crosses the threshold.

A concrete example that reproduces every time: the `writing-plans` skill from the Superpowers plugin has the model write a complete implementation plan in a single `Write` call. The resulting document runs to about 2,000 lines, which lands right above the 31,500-token line — so it stalls on essentially every attempt, and the client aborts before the (otherwise successful) response arrives.

The general shape: **one tool call whose argument exceeds ~31,500 output tokens will exceed a 300s client inactivity timeout.**

## What would fix it

Forwarding `input_json_delta` incrementally, the way `text_delta` and `thinking_delta` are already forwarded, would remove the silence entirely — the events exist and are correctly sequenced, they just arrive together at the end.

Failing that, emitting `ping` events during the gap would at least keep inactivity-based clients alive. Currently zero arrive.

## Environment

- Endpoint: `POST /v1/messages` (Anthropic Messages API shape), `stream: true`
- Model: `claude-opus-5`
- `anthropic-beta: claude-code-20250219,context-1m-2025-08-07,context-management-2025-06-27`
- Prompt: 348,483 input tokens, 294 tool definitions
- Observed on 2026-07-29

Happy to supply the full event-by-event timeline for either run.

Contributor guide

Open the contributing guide

Research direction

Start at the /v1/messages streaming entry point and reproduce the large tool-argument case described in the issue, comparing input_json_delta with text_delta and thinking_delta timing. No source files or tests are named, so trace the stream handling and add or update coverage for incremental tool-argument delivery or keepalive behavior. Done means large and truncated tool calls no longer leave the client silent until completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.