ADORSYS-GIS / ADORSYS-GIS/ai-helm

Envoy AI Gateway: Anthropic streaming reports input_tokens=0 in message_start, breaking client context accounting

Đang mở
#1,072 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Go Template
Star
3
Fork
1
Merge trung bình
19 giờ 24 phút
Pull request đã merge (30 ngày)
80

Mô tả

## Summary

On `api.ai.camer.digital`, the Anthropic-compatible streaming endpoint emits `message_start.usage.input_tokens = 0` and delivers the real prompt size late, inside `message_delta`. Per the Anthropic streaming contract, `message_start` carries the input token count and `message_delta` carries **output** tokens only.

A client that reads the documented field gets zero on every turn. Observed symptom: Claude Code's context-window indicator shows `0 / 1M tokens · 0%` and never advances, while the *total* (from gateway model discovery) is correct.

## Reproduction

Against `api.ai.camer.digital` with a valid bearer, model `qwen3-5-2b-local` (priced at 0, so this costs nothing):

```bash
curl -s -N https://api.ai.camer.digital/anthropic/v1/messages \
-H "authorization: Bearer $(governance-auth token)" \
-H "content-type: application/json" \
-H "anthropic-version: 2023-06-01" \
-d '{"model":"qwen3-5-2b-local","max_tokens":8,"stream":true,
"messages":[{"role":"user","content":"hi"}]}'
```

**Observed:**

```
message_start: usage={"input_tokens": 0, "output_tokens": 0} ← wrong
message_delta: usage={"input_tokens": 13, "output_tokens": 8} ← input arrives here
```

**Expected** (Anthropic contract):

```
message_start: usage={"input_tokens": 13, "output_tokens": 0, "cache_read_input_tokens": …}
message_delta: usage={"output_tokens": 8}
```

## Non-streaming is correct — the bug is streaming-only

The same request without `"stream": true` returns the right numbers:

```json
"usage": {"input_tokens": 13, "output_tokens": 4,
"cache_creation_input_tokens": 0, "cache_read_input_tokens": 0}
```

## Likely mechanism

`lightbridge-governance`'s ADR-0010 records this deployment as `/anthropic/v1/messages` **routing to a `schema: OpenAI` backend**. OpenAI-style streaming reports usage only in the *final* chunk, so a translator emitting the Anthropic wire format has no token count available when it must write `message_start` — it emits zeros and back-fills later. Non-streaming works because the whole upstream response is in hand before anything is serialised.

Every model in the catalogue is OpenAI-shaped upstream, so I could not A/B this against a natively-Anthropic backend to confirm — see *Not verified*.

## Secondary finding: cache fields are dropped when streaming

`cache_creation_input_tokens` and `cache_read_input_tokens` are present non-streaming and **absent** from both streaming usage objects. Any client-side prompt-cache accounting is therefore invisible on the streaming path, which is the path real clients use.

## Impact

- Client-side context accounting is broken for every Anthropic-protocol consumer of this gateway (Claude Code being the one observed).
- It affects the path that matters: interactive clients stream.
- Worth checking whether **server-side** cost accounting reads streaming usage from the same place. If it reads `message_start`, it shares the blind spot; if it computes from the final chunk it is fine — but that should be confirmed rather than assumed, given cost is computed from token counts here.

## Suggested fix

Have the Anthropic translator count prompt tokens itself and emit them in `message_start.usage.input_tokens` rather than deferring to the upstream's final chunk, and pass `cache_creation_input_tokens` / `cache_read_input_tokens` through on the streaming path.

## Not verified

- Claude Code's parser is closed source; that it reads `message_start` is a strong inference from the documented protocol and the matching symptom, not something I confirmed in its code.
- Not A/B tested against a natively-Anthropic backend, since the catalogue has none.
- Tested with one model (`qwen3-5-2b-local`). I did not check whether the behaviour varies per backend.

## AI Usage Declaration

AI-assisted. Claude Opus 5 measured the streaming and non-streaming responses directly against the live gateway and diagnosed the deviation; a human owns intent, verification and consequences. Every number above is captured output, not recalled. https://adorsys-gis.github.io/ai-governance/

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Start at the /anthropic/v1/messages streaming endpoint and consult ADR-0010, which records the OpenAI-shaped backend routing. Compare streaming and non-streaming usage handling, then verify that message_start reports input and cache tokens while message_delta reports output tokens only.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
helm, kubernetes
Lĩnh vực
ai, api, backend
Loại issue
Lỗi
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
45/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.