vllm-project / vllm-project/agentic-api

Persist reliable Responses continuation checkpoints

Open
#71 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
284
Forks
74
Avg merge
1d 17h
Merged PRs (30d)
93

Description

Parent tracker: #69
Design and measured evidence: #65

Summary

Make Responses and Conversation continuation checkpoints reliable enough to carry certified prefix/artifact metadata without promoting incorrect state.

This is the second milestone, after exact llm-d KV-affine routing is available. Correct routing can operate on ordinary rehydrated full requests; durable token artifacts must not be introduced until checkpoint correctness is defined.

Scope

  • Persist immutable execution fingerprint, lineage, renderer-safe boundary proof, token-artifact identity, router-receipt identity, status, and expiration with the accepted response.
  • Bind identity to model weights, tokenizer, renderer/template/parser configuration, instructions, tools, cache_salt, LoRA, multimodal inputs, and format versions.
  • Fail closed for missing, corrupt, unknown, out-of-order, or partially rehydrated certified history.
  • Atomically commit logical output and checkpoint metadata; streaming completion must not imply durable promotion before persistence succeeds.
  • Add unique conversation ordering plus compare-and-swap or serialization for concurrent turns.
  • Define cancellation, retry, and idempotency behavior so duplicate tool effects or checkpoints cannot be promoted.
  • Carry ephemeral prefix/artifact state through gateway-tool subturns and promote only the accepted outer result.
  • Measure checkpoint lookup, validation, and persistence overhead separately from vLLM latency.

Acceptance criteria

  • Cancellation, retry, concurrent turns, corrupt history, and out-of-order items cannot promote an invalid checkpoint or duplicate tool side effects.
  • Rehydration either reconstructs the complete certified model-visible history or returns a typed fallback/failure; it does not silently omit unknown items.
  • Logical output and continuation metadata become durable atomically.
  • Fingerprint changes invalidate reuse without invalidating the logical conversation fallback.
  • Tests cover tool-loop intermediate state, streaming disconnects, persistence failure, and conversation races.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading parent tracker #69 and the design and measured evidence in #65. Trace the checkpoint lifecycle described in the Scope, then use the acceptance criteria to define completion: atomic durable output and metadata, safe rehydration and invalidation, and tests for cancellation, retries, races, corrupt history, tool loops, disconnects, and persistence failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.