ADORSYS-GIS / ADORSYS-GIS/ai-helm
Envoy AI Gateway: Anthropic streaming reports input_tokens=0 in message_start, breaking client context accounting
- Lenguaje dominante
- Go Template
- Estrellas
- 3
- Forks
- 1
- Merge medio
- 19 h 24 min
- PR fusionados (30 d)
- 80
Descripción
## Summary
On `api.ai.camer.digital`, the Anthropic-compatible streaming endpoint emits `message_start.usage.input_tokens = 0` and delivers the real prompt size late, inside `message_delta`. Per the Anthropic streaming contract, `message_start` carries the input token count and `message_delta` carries **output** tokens only.
A client that reads the documented field gets zero on every turn. Observed symptom: Claude Code's context-window indicator shows `0 / 1M tokens · 0%` and never advances, while the *total* (from gateway model discovery) is correct.
## Reproduction
Against `api.ai.camer.digital` with a valid bearer, model `qwen3-5-2b-local` (priced at 0, so this costs nothing):
```bash
curl -s -N https://api.ai.camer.digital/anthropic/v1/messages \
-H "authorization: Bearer $(governance-auth token)" \
-H "content-type: application/json" \
-H "anthropic-version: 2023-06-01" \
-d '{"model":"qwen3-5-2b-local","max_tokens":8,"stream":true,
"messages":[{"role":"user","content":"hi"}]}'
```
**Observed:**
```
message_start: usage={"input_tokens": 0, "output_tokens": 0} ← wrong
message_delta: usage={"input_tokens": 13, "output_tokens": 8} ← input arrives here
```
**Expected** (Anthropic contract):
```
message_start: usage={"input_tokens": 13, "output_tokens": 0, "cache_read_input_tokens": …}
message_delta: usage={"output_tokens": 8}
```
## Non-streaming is correct — the bug is streaming-only
The same request without `"stream": true` returns the right numbers:
```json
"usage": {"input_tokens": 13, "output_tokens": 4,
"cache_creation_input_tokens": 0, "cache_read_input_tokens": 0}
```
## Likely mechanism
`lightbridge-governance`'s ADR-0010 records this deployment as `/anthropic/v1/messages` **routing to a `schema: OpenAI` backend**. OpenAI-style streaming reports usage only in the *final* chunk, so a translator emitting the Anthropic wire format has no token count available when it must write `message_start` — it emits zeros and back-fills later. Non-streaming works because the whole upstream response is in hand before anything is serialised.
Every model in the catalogue is OpenAI-shaped upstream, so I could not A/B this against a natively-Anthropic backend to confirm — see *Not verified*.
## Secondary finding: cache fields are dropped when streaming
`cache_creation_input_tokens` and `cache_read_input_tokens` are present non-streaming and **absent** from both streaming usage objects. Any client-side prompt-cache accounting is therefore invisible on the streaming path, which is the path real clients use.
## Impact
- Client-side context accounting is broken for every Anthropic-protocol consumer of this gateway (Claude Code being the one observed).
- It affects the path that matters: interactive clients stream.
- Worth checking whether **server-side** cost accounting reads streaming usage from the same place. If it reads `message_start`, it shares the blind spot; if it computes from the final chunk it is fine — but that should be confirmed rather than assumed, given cost is computed from token counts here.
## Suggested fix
Have the Anthropic translator count prompt tokens itself and emit them in `message_start.usage.input_tokens` rather than deferring to the upstream's final chunk, and pass `cache_creation_input_tokens` / `cache_read_input_tokens` through on the streaming path.
## Not verified
- Claude Code's parser is closed source; that it reads `message_start` is a strong inference from the documented protocol and the matching symptom, not something I confirmed in its code.
- Not A/B tested against a natively-Anthropic backend, since the catalogue has none.
- Tested with one model (`qwen3-5-2b-local`). I did not check whether the behaviour varies per backend.
## AI Usage Declaration
AI-assisted. Claude Opus 5 measured the streaming and non-streaming responses directly against the live gateway and diagnosed the deviation; a human owns intent, verification and consequences. Every number above is captured output, not recalled. https://adorsys-gis.github.io/ai-governance/
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.