vllm-project / vllm-project/agentic-api
Add portable vLLM token artifacts for Responses continuations
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 284
- Forks
- 74
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 93
Description
Parent tracker: #69
Design and measured evidence: #65
Summary
Add portable, validated token artifacts for Responses continuations after exact llm-d routing and reliable checkpoint persistence are established.
The artifact path is a transport/rendering optimization. It must preserve the KV-cache hit ratio achieved by exact routing and must not depend on a process-local handle that is valid only before llm-d selects a replica.
Scope
- Accept a private portable artifact reference or certified prompt tokens in vLLM and feed the resolved complete token sequence into the existing input/APC path.
- Replace unbounded process-local prompt dictionaries with a bounded, tenant-scoped, expiring local cache backed by an authenticated portable artifact service or inline reseed path.
- Bind artifacts and append certificates to model, tokenizer, renderer/template/parser, instructions, tools,
cache_salt, LoRA, multimodal identity, and expiration. - Define typed artifact miss, expiry, validation failure, and reseed behavior with safe fallback to ordinary full Responses rendering.
- Accumulate all generated streaming token IDs and preserve tool-round boundaries before optionally advancing an artifact.
- Coordinate llm-d placement with authenticated router receipts or server-owned artifact resolution; endpoint-local shortcuts may only be applied after endpoint selection.
- Avoid returning full prompt-token arrays to public clients or creating a second KV-cache namespace outside vLLM APC.
Acceptance criteria
- Any selected compatible replica can resolve the artifact or safely reseed from authenticated material.
- Artifact miss, expiry, restart, wrong-pod selection, and validation failure deterministically fall back without changing model-visible token IDs.
- Portable artifact execution produces the same vLLM APC lookup identity as ordinary full rendering.
- Cache-hit ratio remains equal to the exact-routing baseline while long-context request transfer, reconstruction, rendering/tokenization work, or TTFT improves.
- Local artifact storage is bounded, expiring, tenant-isolated, observable, and safe under process rotation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with parent tracker #69 and the design and measured evidence in #65. Map the vLLM APC path, llm-d placement, artifact validation, reseeding, and streaming token boundaries before proposing an implementation. Done means compatible replicas resolve or safely reseed artifacts, fallbacks preserve token IDs and APC identity, and bounded tenant-scoped storage meets the listed acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend-api-design, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100