github / github/copilot-cli

Streaming: tool_use input_json_delta is buffered until complete, causing multi-minute silences on large tool arguments

未关闭
#4,286 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
area:models area:networking
主要语言
Shell
星标
11.2k
派生
1.9k
平均合并
14 小时 16 分钟
30 天内合并 PR
6

描述

## Summary

On `/v1/messages` streaming responses, `input_json_delta` events for a `tool_use` block appear to be buffered until the tool-argument JSON is complete, then flushed all at once. For a large tool argument this produces a multi-minute window in which the connection is open and healthy but carries **zero bytes**.

Text and thinking blocks in the same response stream normally. Only the tool-argument deltas are withheld.

The request ultimately succeeds — the model was working the whole time — but by then any client with an inactivity timeout has already given up. Claude Code aborts at 300s of no content events with `Response stalled mid-stream`, so any single tool call above roughly 31,500 output tokens fails deterministically.

No `ping` events are emitted during the silence.

## Measurements

Same request replayed twice against `/v1/messages`, `claude-opus-5`, streaming, varying only `max_tokens`:

| `max_tokens` | `output_tokens` | longest silence | ms per output token | tool args delivered | `stop_reason` | wall clock |
|---|---|---|---|---|---|---|
| 64000 | 35,899 | **329.5s** | 9.18 | 85,744 chars | `tool_use` | 339.5s |
| 8000 | 8,000 | **78.3s** | 9.79 | **0 chars** | `max_tokens` | 84.4s |

Two independent measurements agree on ms-per-output-token within 6%, so the silence scales linearly with the size of the tool argument being generated:

```
silence ≈ output_tokens × 9.5ms (≈105 tokens/sec)
300s client threshold → ≈31,500 output tokens
```

## Event timeline — 64000 case

```
t= 3.776 message_start
t= 6.488 content_block_delta thinking_delta ← streams normally
t= 6.648 content_block_delta text_delta ← streams normally
t= 6.649 content_block_start tool_use "Write"
t= 6.649 content_block_delta input_json_delta partial_json:"" ← empty
──────────────── 329.5s, zero bytes ────────────────
t=336.140 content_block_delta input_json_delta partial_json:"{\""
t=336.140 …12,215 deltas totalling 85,744 chars…
t=339.410 content_block_stop
t=339.441 message_delta stop_reason: tool_use
t=339.442 message_stop
```

Time to first byte was 3.8s, so the connection itself was fine.

The 85,744 characters arrive in **3.3 seconds** after the silence ends — about 26,000 chars/sec, an order of magnitude faster than the model's own generation rate. That gap is what points at buffering rather than slow generation: the content was produced steadily over 329.5s and released in one burst at the end.

## Secondary issue: truncated tool JSON delivers nothing but bills in full

In the `max_tokens=8000` run the tool-argument JSON was cut off by the token limit. Because the JSON never completed, **not a single character was forwarded**:

```
t= 6.129 content_block_start tool_use "Write"
t= 6.129 content_block_delta input_json_delta partial_json:""
────────── 78.3s silence ──────────
t=84.400 message_delta stop_reason: max_tokens
t=84.400 message_stop
```

The response is `200`, `usage.output_tokens` is 8,000, and the caller receives an empty tool call. All-or-nothing delivery means a caller pays full price for zero usable output whenever a tool argument hits the token limit. Incremental forwarding would at least surface the partial JSON.

## Impact

This is reproducible and not rare — it is a function of output size, not of load or luck. Any workflow that has the model emit one large artifact through a single tool call crosses the threshold.

A concrete example that reproduces every time: the `writing-plans` skill from the Superpowers plugin has the model write a complete implementation plan in a single `Write` call. The resulting document runs to about 2,000 lines, which lands right above the 31,500-token line — so it stalls on essentially every attempt, and the client aborts before the (otherwise successful) response arrives.

The general shape: **one tool call whose argument exceeds ~31,500 output tokens will exceed a 300s client inactivity timeout.**

## What would fix it

Forwarding `input_json_delta` incrementally, the way `text_delta` and `thinking_delta` are already forwarded, would remove the silence entirely — the events exist and are correctly sequenced, they just arrive together at the end.

Failing that, emitting `ping` events during the gap would at least keep inactivity-based clients alive. Currently zero arrive.

## Environment

- Endpoint: `POST /v1/messages` (Anthropic Messages API shape), `stream: true`
- Model: `claude-opus-5`
- `anthropic-beta: claude-code-20250219,context-1m-2025-08-07,context-management-2025-06-27`
- Prompt: 348,483 input tokens, 294 tool definitions
- Observed on 2026-07-29

Happy to supply the full event-by-event timeline for either run.

贡献指南

打开贡献指南

调研方向

从 /v1/messages 流式入口开始,重现 issue 中描述的大型工具参数案例,并比较 input_json_delta 与 text_delta 和 thinking_delta 的时序。没有指定源文件或测试,因此请跟踪流处理,并为工具参数的增量传递或 keepalive 行为添加或更新覆盖。完成的标志是,大型和被截断的工具调用不再让客户端一直沉默到完成为止。

由索引模型根据 Issue 内容生成。

评估

技术栈
shell
领域
api, backend
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
冷清
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。