vllm-project / vllm-project/agentic-api

Add bounded, validated Messages streaming and tool scheduling

Open
#313 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement enterprise-readiness
Dominant language
Rust
Stars
284
Forks
74
Avg merge
1d 17h
Merged PRs (30d)
93

Description

Problem statement / motivation

Enterprise deployments need predictable memory use and explicit completion semantics for Messages requests that execute built-in tools. Extend the existing native Messages loop with a documented lifecycle and resource contract across all inference rounds.

Proposed solution
  • Validate message and content-block transitions through one Messages ingestion path, keeping transport framing and client delivery separate from semantic validation.
  • Bound retained text, thinking, signatures, tool arguments, tool call outputs, pending calls, and reconstructed history. Distinguish retained bytes from wire bytes so smaller deltas do not consume disproportionate response capacity.
  • Reuse bounded scheduling contracts for built-in tools, with defined timeout, overload, disconnect, cancellation, and task-cleanup behavior.
  • Preserve native Messages content, client-executed function tools, rebased indexes, and one public message lifecycle.
Acceptance criteria
  • Invalid ordering, duplicate indexes/completions, missing identifiers, malformed event data, and open blocks at termination have explicit tested outcomes.
  • Premature EOF cannot dispatch an incomplete tool call or become an apparently successful message. Document any intentional engine-specific tolerance; retain forward compatibility for harmless unknown events.
  • Limits cover accumulated content and pending work across rounds, with deterministic errors and bounded resource release.
  • Tests cover fragmented UTF-8/SSE, many small deltas, large arguments/outputs, slow readers, disconnects during inference/tools, and round-limit exhaustion.
  • Preserve the message_stop completion and forced-tool-choice behavior shipped in #290 and #296.
Additional context

Enterprise Readiness enhancement. Coordinate with #241/#244 and #288 for shared boundary and budget contracts; this issue owns Messages behavior. No worker-placement change is required without measurements.

Source review at afa6c14, not a new runtime reproduction: Messages streaming and Messages JSON loop.

Enterprise Readiness tracker: https://github.com/vllm-project/agentic-api/issues/316

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading crates/agentic-server-core/src/executor/messages_stream.rs and messages_loop.rs at commit afa6c14, then review the shared boundary and budget contracts in issues #241, #244, and #288. Define the Messages lifecycle and resource behavior before implementing it; done means the listed ordering, EOF, limit, cancellation, streaming, tool, and round-limit cases have deterministic tests while preserving message_stop and forced-tool-choice behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.