api7 / api7/aisix

ensemble streaming usage: fold panel sum once on a synthesized terminal frame (multi-usage-chunk judge over-count)

Open
#617 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
157
Forks
32
Avg merge
1h 25m
Merged PRs (30d)
145

Description

Follow-up to #614 / PR #616 (independent-audit HIGH).

Summary

A streaming ensemble folds the panel's usage sum (base_usage) into the client-facing terminal usage chunk via build_sse_stream. The fold is applied per forwarded chunk that carries usage (crates/aisix-proxy/src/chat.rs, the chunk.usage.as_mut() block). This is exact when the judge upstream emits usage on a single terminal frame — true for the OpenAI / Anthropic / DeepSeek bridges (the gateway injects include_usage, yielding one usage-only terminal chunk). It over-applies when the judge emits usage on more than one forwarded chunk.

Reachable case

The Vertex/Gemini bridge maps usageMetadataChatChunk.usage on every chunk that carries it (crates/aisix-provider-vertex/src/bridge.rs), and a Gemini model is a valid streaming-ensemble judge. With such a judge + a client that requested stream_options.include_usage, the panel sum is added to each usage-bearing frame. A client that sums usage across frames over-reports by the panel cost × (frames − 1). (A client that reads only the terminal frame is unaffected when the upstream usage is cumulative, but the wire output is still non-conformant.)

The comp accumulator in the same function already guards this class with "max, not last" (see its comment); the base_usage fold has no equivalent.

Fix direction

Apply the panel sum once, to a single terminal usage frame, instead of per-chunk:

  • For an ensemble (base_usage != 0): strip usage from forwarded chunks (after comp captures it) and emit one synthesized terminal usage-only frame = comp + base_usage before [DONE].
  • Keep the single-upstream path byte-for-byte unchanged (base_usage == 0 ⇒ no synthesis, no strip).

Scope / impact

Narrow: streaming ensemble and a multi-usage-chunk judge (Gemini/Vertex) and client-requested include_usage. Non-streaming and single-terminal-frame judges (OpenAI/Anthropic/DeepSeek) are already exact (PR #616). Not a correctness/safety issue — usage-reporting accuracy only. Tracked separately so it doesn't block the #614 core fix.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in crates/aisix-proxy/src/chat.rs, at build_sse_stream and the chunk.usage.as_mut() block; compare the base_usage fold with the comp accumulator. Check crates/aisix-provider-vertex/src/bridge.rs to understand the multi-usage-chunk input. Done when an ensemble emits one synthesized terminal usage-only frame before [DONE], while the single-upstream path remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, rust
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.