NVIDIA / NVIDIA/TensorRT-LLM

[Feature]: General cross-instance KV-cache transfer/staging (kv_transfer)

Open
#15,735 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Disaggregated serving
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

🚀 The feature, motivation and pitch

Disaggregated serving already moves a request's KV from a context server to a generation server on
demand. We'd like to expose that same transceiver path as a standalone, proactive operation: tell
instance B to pull a given prompt's KV from instance A and commit it to B's reuse cache, without
generating
.

Motivation. Today the only ways to get a prompt's KV onto a specific instance are to actually run
a request there (cold prefill) or to rely on on-demand disagg at request time. A proactive transfer
enables, all on the existing transceiver:

  • Cache migration / rebalancing — move a hot prompt's KV off an overloaded instance onto an idle
    one, instead of recomputing it there.
  • Replication — keep a copy of a prompt's KV on more than one instance.

Proposed surface (additive, off by default):

  • A kv_transfer_only flag on a generation_only disaggregated request: the engine pulls the KV over
    the transceiver, commits it for reuse, and finishes without a generation forward pass.
  • A fire-and-forget POST /v1/kv_transfer {model, messages|prompt, source} endpoint (+
    GET /v1/kv_transfer/stats) that orchestrates the existing context_onlygeneration_only ticket
    flow internally, so callers never handle disagg tickets.

It reuses the existing disagg transceiver — no new transport, and regular disagg/serving paths are
unchanged. A PR follows in #15736.

Alternatives

No response

Additional context

No response

Before submitting a new issue...
  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review PR #15736 first, then trace the existing context_only → generation_only ticket flow and disagg transceiver path. The work is done when the additive kv_transfer_only behavior and the POST /v1/kv_transfer plus GET /v1/kv_transfer/stats surfaces support proactive KV staging without a generation pass, while regular disaggregated serving remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai-infra-agents, api, backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.