ensemble streaming usage: fold panel sum once on a synthesized terminal frame (multi-usage-chunk judge over-count)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 157
- Forks
- 32
- Avg merge
- 1h 25m
- Merged PRs (30d)
- 145
Description
Follow-up to #614 / PR #616 (independent-audit HIGH).
Summary
A streaming ensemble folds the panel's usage sum (base_usage) into the client-facing terminal usage chunk via build_sse_stream. The fold is applied per forwarded chunk that carries usage (crates/aisix-proxy/src/chat.rs, the chunk.usage.as_mut() block). This is exact when the judge upstream emits usage on a single terminal frame — true for the OpenAI / Anthropic / DeepSeek bridges (the gateway injects include_usage, yielding one usage-only terminal chunk). It over-applies when the judge emits usage on more than one forwarded chunk.
Reachable case
The Vertex/Gemini bridge maps usageMetadata → ChatChunk.usage on every chunk that carries it (crates/aisix-provider-vertex/src/bridge.rs), and a Gemini model is a valid streaming-ensemble judge. With such a judge + a client that requested stream_options.include_usage, the panel sum is added to each usage-bearing frame. A client that sums usage across frames over-reports by the panel cost × (frames − 1). (A client that reads only the terminal frame is unaffected when the upstream usage is cumulative, but the wire output is still non-conformant.)
The comp accumulator in the same function already guards this class with "max, not last" (see its comment); the base_usage fold has no equivalent.
Fix direction
Apply the panel sum once, to a single terminal usage frame, instead of per-chunk:
- For an ensemble (
base_usage != 0): stripusagefrom forwarded chunks (aftercompcaptures it) and emit one synthesized terminal usage-only frame =comp + base_usagebefore[DONE]. - Keep the single-upstream path byte-for-byte unchanged (
base_usage == 0⇒ no synthesis, no strip).
Scope / impact
Narrow: streaming ensemble and a multi-usage-chunk judge (Gemini/Vertex) and client-requested include_usage. Non-streaming and single-terminal-frame judges (OpenAI/Anthropic/DeepSeek) are already exact (PR #616). Not a correctness/safety issue — usage-reporting accuracy only. Tracked separately so it doesn't block the #614 core fix.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in crates/aisix-proxy/src/chat.rs, at build_sse_stream and the chunk.usage.as_mut() block; compare the base_usage fold with the comp accumulator. Check crates/aisix-provider-vertex/src/bridge.rs to understand the multi-usage-chunk input. Done when an ensemble emits one synthesized terminal usage-only frame before [DONE], while the single-upstream path remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, rust
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100