ADORSYS-GIS / ADORSYS-GIS/ai-helm

Envoy AI Gateway: Anthropic streaming reports input_tokens=0 in message_start, breaking client context accounting

Abierto
#1,072 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Go Template
Estrellas
3
Forks
1
Merge medio
19 h 24 min
PR fusionados (30 d)
80

Descripción

## Summary

On `api.ai.camer.digital`, the Anthropic-compatible streaming endpoint emits `message_start.usage.input_tokens = 0` and delivers the real prompt size late, inside `message_delta`. Per the Anthropic streaming contract, `message_start` carries the input token count and `message_delta` carries **output** tokens only.

A client that reads the documented field gets zero on every turn. Observed symptom: Claude Code's context-window indicator shows `0 / 1M tokens · 0%` and never advances, while the *total* (from gateway model discovery) is correct.

## Reproduction

Against `api.ai.camer.digital` with a valid bearer, model `qwen3-5-2b-local` (priced at 0, so this costs nothing):

```bash
curl -s -N https://api.ai.camer.digital/anthropic/v1/messages \
-H "authorization: Bearer $(governance-auth token)" \
-H "content-type: application/json" \
-H "anthropic-version: 2023-06-01" \
-d '{"model":"qwen3-5-2b-local","max_tokens":8,"stream":true,
"messages":[{"role":"user","content":"hi"}]}'
```

**Observed:**

```
message_start: usage={"input_tokens": 0, "output_tokens": 0} ← wrong
message_delta: usage={"input_tokens": 13, "output_tokens": 8} ← input arrives here
```

**Expected** (Anthropic contract):

```
message_start: usage={"input_tokens": 13, "output_tokens": 0, "cache_read_input_tokens": …}
message_delta: usage={"output_tokens": 8}
```

## Non-streaming is correct — the bug is streaming-only

The same request without `"stream": true` returns the right numbers:

```json
"usage": {"input_tokens": 13, "output_tokens": 4,
"cache_creation_input_tokens": 0, "cache_read_input_tokens": 0}
```

## Likely mechanism

`lightbridge-governance`'s ADR-0010 records this deployment as `/anthropic/v1/messages` **routing to a `schema: OpenAI` backend**. OpenAI-style streaming reports usage only in the *final* chunk, so a translator emitting the Anthropic wire format has no token count available when it must write `message_start` — it emits zeros and back-fills later. Non-streaming works because the whole upstream response is in hand before anything is serialised.

Every model in the catalogue is OpenAI-shaped upstream, so I could not A/B this against a natively-Anthropic backend to confirm — see *Not verified*.

## Secondary finding: cache fields are dropped when streaming

`cache_creation_input_tokens` and `cache_read_input_tokens` are present non-streaming and **absent** from both streaming usage objects. Any client-side prompt-cache accounting is therefore invisible on the streaming path, which is the path real clients use.

## Impact

- Client-side context accounting is broken for every Anthropic-protocol consumer of this gateway (Claude Code being the one observed).
- It affects the path that matters: interactive clients stream.
- Worth checking whether **server-side** cost accounting reads streaming usage from the same place. If it reads `message_start`, it shares the blind spot; if it computes from the final chunk it is fine — but that should be confirmed rather than assumed, given cost is computed from token counts here.

## Suggested fix

Have the Anthropic translator count prompt tokens itself and emit them in `message_start.usage.input_tokens` rather than deferring to the upstream's final chunk, and pass `cache_creation_input_tokens` / `cache_read_input_tokens` through on the streaming path.

## Not verified

- Claude Code's parser is closed source; that it reads `message_start` is a strong inference from the documented protocol and the matching symptom, not something I confirmed in its code.
- Not A/B tested against a natively-Anthropic backend, since the catalogue has none.
- Tested with one model (`qwen3-5-2b-local`). I did not check whether the behaviour varies per backend.

## AI Usage Declaration

AI-assisted. Claude Opus 5 measured the streaming and non-streaming responses directly against the live gateway and diagnosed the deviation; a human owns intent, verification and consequences. Every number above is captured output, not recalled. https://adorsys-gis.github.io/ai-governance/

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Start at the /anthropic/v1/messages streaming endpoint and consult ADR-0010, which records the OpenAI-shaped backend routing. Compare streaming and non-streaming usage handling, then verify that message_start reports input and cache tokens while message_delta reports output tokens only.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
helm, kubernetes
Área
ai, api, backend
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Activo
Claridad
Bastante claro
Aptitud para principiantes
45/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.