vllm-project / vllm-project/agentic-api
Add complete multi-round Messages usage accounting
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 284
- Forks
- 74
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 93
Description
Problem statement / motivation
Operators need the full inference cost of a Messages turn, including rounds used by built-in tools. Provide consistent, documented accounting for JSON and SSE clients without counting repeated usage snapshots more than once.
Proposed solution
- Add a typed per-round usage accumulator and a turn aggregate shared by the JSON and streaming Messages paths.
- Define input, output, cache-read, and cache-creation accounting for supported upstream fields. Distinguish missing usage from zero and retain documented provider semantics.
- Specify how complete turn usage is exposed while preserving the native Messages streaming schema. If the wire protocol cannot represent every total, document a separate typed telemetry surface rather than inventing wire fields.
- Separate per-round records from the aggregate so downstream metering counts each inference contribution once.
Acceptance criteria
- Deterministic one-, two-, and multi-tool-round fixtures verify totals for JSON and SSE.
- Tests cover initial/terminal usage snapshots, repeated cumulative counters, absent/partial usage, cache counters, and unknown supported extensions.
- JSON and SSE expose equivalent complete-turn accounting through the documented surfaces.
- Errors, cancellation, client-tool handoff, and round-limit exhaustion report available usage as partial when appropriate; they never fabricate a completed total.
- Usage from hidden built-in-tool rounds is included without exposing their internal content or double-counting the public aggregate.
- Document units, accounting boundaries, and examples for operators.
Additional context
Enterprise Readiness enhancement. Coordinate with #279 for OpenTelemetry export; this issue owns usage semantics and correctness, not a billing platform.
Source review at afa6c14, not a new runtime reproduction: Messages streaming and Messages JSON loop.
Enterprise Readiness tracker: https://github.com/vllm-project/agentic-api/issues/316
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the referenced messages_stream.rs and messages_loop.rs at commit afa6c14, then trace how JSON and SSE paths represent usage across inference rounds. Add deterministic one-, two-, and multi-tool-round fixtures covering the listed usage cases, and document the accounting units, boundaries, and exposure surfaces. Done means JSON and SSE provide equivalent complete-turn accounting without double-counting, while partial outcomes remain marked partial.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100