vllm-project / vllm-project/agentic-api

Add portable vLLM token artifacts for Responses continuations

Open
#70 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
284
Forks
74
Avg merge
1d 17h
Merged PRs (30d)
93

Description

Parent tracker: #69
Design and measured evidence: #65

Summary

Add portable, validated token artifacts for Responses continuations after exact llm-d routing and reliable checkpoint persistence are established.

The artifact path is a transport/rendering optimization. It must preserve the KV-cache hit ratio achieved by exact routing and must not depend on a process-local handle that is valid only before llm-d selects a replica.

Scope

  • Accept a private portable artifact reference or certified prompt tokens in vLLM and feed the resolved complete token sequence into the existing input/APC path.
  • Replace unbounded process-local prompt dictionaries with a bounded, tenant-scoped, expiring local cache backed by an authenticated portable artifact service or inline reseed path.
  • Bind artifacts and append certificates to model, tokenizer, renderer/template/parser, instructions, tools, cache_salt, LoRA, multimodal identity, and expiration.
  • Define typed artifact miss, expiry, validation failure, and reseed behavior with safe fallback to ordinary full Responses rendering.
  • Accumulate all generated streaming token IDs and preserve tool-round boundaries before optionally advancing an artifact.
  • Coordinate llm-d placement with authenticated router receipts or server-owned artifact resolution; endpoint-local shortcuts may only be applied after endpoint selection.
  • Avoid returning full prompt-token arrays to public clients or creating a second KV-cache namespace outside vLLM APC.

Acceptance criteria

  • Any selected compatible replica can resolve the artifact or safely reseed from authenticated material.
  • Artifact miss, expiry, restart, wrong-pod selection, and validation failure deterministically fall back without changing model-visible token IDs.
  • Portable artifact execution produces the same vLLM APC lookup identity as ordinary full rendering.
  • Cache-hit ratio remains equal to the exact-routing baseline while long-context request transfer, reconstruction, rendering/tokenization work, or TTFT improves.
  • Local artifact storage is bounded, expiring, tenant-isolated, observable, and safe under process rotation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with parent tracker #69 and the design and measured evidence in #65. Map the vLLM APC path, llm-d placement, artifact validation, reseeding, and streaming token boundaries before proposing an implementation. Done means compatible replicas resolve or safely reseed artifacts, fallbacks preserve token IDs and APC identity, and bounded tenant-scoped storage meets the listed acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Domain
backend-api-design, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.