vllm-project / vllm-project/agentic-api
Add bounded, validated Messages streaming and tool scheduling
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 284
- Forks
- 74
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 93
Description
Problem statement / motivation
Enterprise deployments need predictable memory use and explicit completion semantics for Messages requests that execute built-in tools. Extend the existing native Messages loop with a documented lifecycle and resource contract across all inference rounds.
Proposed solution
- Validate message and content-block transitions through one Messages ingestion path, keeping transport framing and client delivery separate from semantic validation.
- Bound retained text, thinking, signatures, tool arguments, tool call outputs, pending calls, and reconstructed history. Distinguish retained bytes from wire bytes so smaller deltas do not consume disproportionate response capacity.
- Reuse bounded scheduling contracts for built-in tools, with defined timeout, overload, disconnect, cancellation, and task-cleanup behavior.
- Preserve native Messages content, client-executed function tools, rebased indexes, and one public message lifecycle.
Acceptance criteria
- Invalid ordering, duplicate indexes/completions, missing identifiers, malformed event data, and open blocks at termination have explicit tested outcomes.
- Premature EOF cannot dispatch an incomplete tool call or become an apparently successful message. Document any intentional engine-specific tolerance; retain forward compatibility for harmless unknown events.
- Limits cover accumulated content and pending work across rounds, with deterministic errors and bounded resource release.
- Tests cover fragmented UTF-8/SSE, many small deltas, large arguments/outputs, slow readers, disconnects during inference/tools, and round-limit exhaustion.
- Preserve the message_stop completion and forced-tool-choice behavior shipped in #290 and #296.
Additional context
Enterprise Readiness enhancement. Coordinate with #241/#244 and #288 for shared boundary and budget contracts; this issue owns Messages behavior. No worker-placement change is required without measurements.
Source review at afa6c14, not a new runtime reproduction: Messages streaming and Messages JSON loop.
Enterprise Readiness tracker: https://github.com/vllm-project/agentic-api/issues/316
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading crates/agentic-server-core/src/executor/messages_stream.rs and messages_loop.rs at commit afa6c14, then review the shared boundary and budget contracts in issues #241, #244, and #288. Define the Messages lifecycle and resource behavior before implementing it; done means the listed ordering, EOF, limit, cancellation, streaming, tool, and round-limit cases have deterministic tests while preserving message_stop and forced-tool-choice behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100