ADORSYS-GIS / ADORSYS-GIS/ai-helm

Envoy AI Gateway: Anthropic streaming reports input_tokens=0 in message_start, breaking client context accounting

Abierto
#1,072 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Go Template
Estrellas
3
Forks
1
Merge medio
19 h 24 min
PR fusionados (30 d)
80

Descripción

## Summary

On `api.ai.camer.digital`, the Anthropic-compatible streaming endpoint emits `message_start.usage.input_tokens = 0` and delivers the real prompt size late, inside `message_delta`. Per the Anthropic streaming contract, `message_start` carries the input token count and `message_delta` carries **output** tokens only.

A client that reads the documented field gets zero on every turn. Observed symptom: Claude Code's context-window indicator shows `0 / 1M tokens · 0%` and never advances, while the *total* (from gateway model discovery) is correct.

## Reproduction

Against `api.ai.camer.digital` with a valid bearer, model `qwen3-5-2b-local` (priced at 0, so this costs nothing):

```bash
curl -s -N https://api.ai.camer.digital/anthropic/v1/messages \
-H "authorization: Bearer $(governance-auth token)" \
-H "content-type: application/json" \
-H "anthropic-version: 2023-06-01" \
-d '{"model":"qwen3-5-2b-local","max_tokens":8,"stream":true,
"messages":[{"role":"user","content":"hi"}]}'
```

**Observed:**

```
message_start: usage={"input_tokens": 0, "output_tokens": 0} ← wrong
message_delta: usage={"input_tokens": 13, "output_tokens": 8} ← input arrives here
```

**Expected** (Anthropic contract):

```
message_start: usage={"input_tokens": 13, "output_tokens": 0, "cache_read_input_tokens": …}
message_delta: usage={"output_tokens": 8}
```

## Non-streaming is correct — the bug is streaming-only

The same request without `"stream": true` returns the right numbers:

```json
"usage": {"input_tokens": 13, "output_tokens": 4,
"cache_creation_input_tokens": 0, "cache_read_input_tokens": 0}
```

## Likely mechanism

`lightbridge-governance`'s ADR-0010 records this deployment as `/anthropic/v1/messages` **routing to a `schema: OpenAI` backend**. OpenAI-style streaming reports usage only in the *final* chunk, so a translator emitting the Anthropic wire format has no token count available when it must write `message_start` — it emits zeros and back-fills later. Non-streaming works because the whole upstream response is in hand before anything is serialised.

Every model in the catalogue is OpenAI-shaped upstream, so I could not A/B this against a natively-Anthropic backend to confirm — see *Not verified*.

## Secondary finding: cache fields are dropped when streaming

`cache_creation_input_tokens` and `cache_read_input_tokens` are present non-streaming and **absent** from both streaming usage objects. Any client-side prompt-cache accounting is therefore invisible on the streaming path, which is the path real clients use.

## Impact

- Client-side context accounting is broken for every Anthropic-protocol consumer of this gateway (Claude Code being the one observed).
- It affects the path that matters: interactive clients stream.
- Worth checking whether **server-side** cost accounting reads streaming usage from the same place. If it reads `message_start`, it shares the blind spot; if it computes from the final chunk it is fine — but that should be confirmed rather than assumed, given cost is computed from token counts here.

## Suggested fix

Have the Anthropic translator count prompt tokens itself and emit them in `message_start.usage.input_tokens` rather than deferring to the upstream's final chunk, and pass `cache_creation_input_tokens` / `cache_read_input_tokens` through on the streaming path.

## Not verified

- Claude Code's parser is closed source; that it reads `message_start` is a strong inference from the documented protocol and the matching symptom, not something I confirmed in its code.
- Not A/B tested against a natively-Anthropic backend, since the catalogue has none.
- Tested with one model (`qwen3-5-2b-local`). I did not check whether the behaviour varies per backend.

## AI Usage Declaration

AI-assisted. Claude Opus 5 measured the streaming and non-streaming responses directly against the live gateway and diagnosed the deviation; a human owns intent, verification and consequences. Every number above is captured output, not recalled. https://adorsys-gis.github.io/ai-governance/

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.