vllm-project / vllm-project/agentic-api

Add complete multi-round Messages usage accounting

Open
#315 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement enterprise-readiness
Dominant language
Rust
Stars
284
Forks
74
Avg merge
1d 17h
Merged PRs (30d)
93

Description

Problem statement / motivation

Operators need the full inference cost of a Messages turn, including rounds used by built-in tools. Provide consistent, documented accounting for JSON and SSE clients without counting repeated usage snapshots more than once.

Proposed solution
  • Add a typed per-round usage accumulator and a turn aggregate shared by the JSON and streaming Messages paths.
  • Define input, output, cache-read, and cache-creation accounting for supported upstream fields. Distinguish missing usage from zero and retain documented provider semantics.
  • Specify how complete turn usage is exposed while preserving the native Messages streaming schema. If the wire protocol cannot represent every total, document a separate typed telemetry surface rather than inventing wire fields.
  • Separate per-round records from the aggregate so downstream metering counts each inference contribution once.
Acceptance criteria
  • Deterministic one-, two-, and multi-tool-round fixtures verify totals for JSON and SSE.
  • Tests cover initial/terminal usage snapshots, repeated cumulative counters, absent/partial usage, cache counters, and unknown supported extensions.
  • JSON and SSE expose equivalent complete-turn accounting through the documented surfaces.
  • Errors, cancellation, client-tool handoff, and round-limit exhaustion report available usage as partial when appropriate; they never fabricate a completed total.
  • Usage from hidden built-in-tool rounds is included without exposing their internal content or double-counting the public aggregate.
  • Document units, accounting boundaries, and examples for operators.
Additional context

Enterprise Readiness enhancement. Coordinate with #279 for OpenTelemetry export; this issue owns usage semantics and correctness, not a billing platform.

Source review at afa6c14, not a new runtime reproduction: Messages streaming and Messages JSON loop.

Enterprise Readiness tracker: https://github.com/vllm-project/agentic-api/issues/316

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the referenced messages_stream.rs and messages_loop.rs at commit afa6c14, then trace how JSON and SSE paths represent usage across inference rounds. Add deterministic one-, two-, and multi-tool-round fixtures covering the listed usage cases, and document the accounting units, boundaries, and exposure surfaces. Done means JSON and SSE provide equivalent complete-turn accounting without double-counting, while partial outcomes remain marked partial.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.