[RFC]: Versioned KV Hints Protocol for TRT-LLM

Motivation.

Related RFC: RFC #18151 — Router Hint initiated P2P KV Cache Transfer Between TRT-LLM Workers

Motivation

RFC #18151 proposes a router-initiated KvHint surface as a first-class construct for e2e programmatic KV management between TRT-LLM and an orchestrator (such as Dynamo). This RFC proposes a concrete implementation of that hint protocol, such that it is extensible, expressive, and durable: the protocol is compatible with various orchestrators, enables semantics-free expression of various types of hints (e.g. fetch, share, …), and guarantees durability via typing and versioning.

Today, integrations (such as Dynamo) prototype TRT-LLM KV hints via indirect, untyped plumbings, such as extra_args and kv_transfer_params. RFC #18151 then introduces a flat, fetch-only kv_hint field. KvHints instead provides a typed, versioned request contract within TRT-LLM which can be accessed by routers, schedulers, cache managers, and cache transceivers:

Router / control plane
  -> KvHintsEnvelope
  -> TRT-LLM request ingress
  -> GenerationRequest.kv_hints
  -> executor Request.kv_hints
  -> LlmRequest.kv_hints
  -> scheduler-visible metadata

This RFC does not replace RFC #18151. That RFC remains the first concrete action: engine prefetch, CacheTransceiver receive, fallback, and block safety. This RFC is only the durable transport.

Proposed Change.

Proposed API

Canonical Types

The canonical transport types — KvHintsEnvelope, KvHintAction — should be broadly accessible and live outside any specific cache manager or transfer backend:

tensorrt_llm/kv_hints/
  __init__.py
  protocol.py

@dataclass(slots=True)
class KvHintsEnvelope:
    protocol_version: str
    message_id: str
    actions: list[KvHintAction]


@dataclass(slots=True)
class KvHintAction:
    action_id: str
    action_type: str
    action_version: str
    payload: dict[str, Any]

Typed Envelope Contents

A KvHintsEnvelope requires:

• A protocol_version
• A message_id
• A list of KvHintAction objects

Each KvHintAction contains:

• action_id
• action_version
• action_type
• An action-specific JSON payload

The envelope and action fields are typed. The payload remains opaque to the transport layer and is validated by the TRT-LLM component implementing that action type. Action-specific payload types should be introduced with their implementations.

Example of a KvHintsEnvelope

The following uses kv.fetch to illustrate the envelope. The payload fields come from RFC #18151; this RFC does not define their semantics.

{
  "protocol_version": "0.1",
  "message_id": "msg-123",
  "actions": [
    {
      "action_id": "a1",
      "action_type": "kv.fetch",
      "action_version": "1.0",
      "payload": {
        "source_control_endpoint": "http://worker-a:8000/v1/data_transceiver_state",
        "block_hashes": [111, 222, 333]
      }
    }
  ]
}

This RFC does not define a closed set of action_type values. Each action type, including its payload schema, validation, and execution semantics, should be introduced with its implementation. kv.fetch above is only an example of how an action sits in the envelope. See RFC #18151 for that action's semantics.

Additional Durability Guarantees

• Envelope and action versions evolve independently.
• Required schema changes bump the relevant version; optional additions and new action types do not require an envelope-version bump.
• Every request-construction path should pass either a KvHintsEnvelope or None. Requests without hints retain existing behavior. Orchestrators, such as Dynamo, construct or deserialize the canonical envelope at TRT-LLM ingress.
• Future TRT-LLM components implementing an action (e.g. scheduler, cache manager, CacheTransceiver) should skip unsupported action versions or invalid payloads without preventing other supported actions in the envelope from being processed. Raw ingress adapters should reject unsupported envelope versions.

KvHints Propagation

KvHints are request metadata, not sampling parameters. They should be available explicitly throughout the TRT-LLM request path.

Schema:

tensorrt_llm/kv_hints/protocol.py
  - KvHintsEnvelope
  - KvHintAction

Then, as the orchestrator and TRT-LLM execute policy-driven KV management, these KvHints become the clean transport layer:

Runtime propagation:
  Dynamo policy produces implementation-specific hints
    -> orchestrator-side TRT-LLM adapter constructs KvHintsEnvelope
    -> TRT-LLM request ingress receives KvHintsEnvelope
    -> LLM.generate_async(kv_hints=...) or equivalent
    -> GenerationRequest.kv_hints
    -> executor Request.kv_hints
    -> LlmRequest.kv_hints
    -> scheduler-visible metadata

This RFC stops at transport. RFC #18151 already defines how a kv.fetch action is executed: context prefetch, CacheTransceiver receive, fallback, and block safety.

Relationship to RFC #18151

The flat first-implementation hint:

KvHint(
    source_control_endpoint="http://worker-a:8000/v1/data_transceiver_state",
    block_hashes=[111, 222, 333],
)

maps to one envelope action:

{
  "action_type": "kv.fetch",
  "action_version": "1.0",
  "payload": {
    "source_control_endpoint": "http://worker-a:8000/v1/data_transceiver_state",
    "block_hashes": [111, 222, 333]
  }
}

Feedback Period.

No response

CC List.

@mkhazraee @ziqifan617 @Tabrizian @kahalon

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.