[RFC]: Semantic KV Cache Reuse

Motivation.

Recent semantic KV research, including SemShareKV (https://arxiv.org/pdf/2509.24832) and work done by WorldFlow AI on SemBlend (https://github.com/WorldFlowAI/semblend) with SGLang around semantic KV cache reuse suggest the field is moving beyond just exact-prefix caching.

Related to:
https://github.com/vllm-project/vllm/issues/44223
https://github.com/ai-dynamo/dynamo/issues/10127
https://github.com/ibifrost/sglang/pull/1

Proposed Change.

Semantic KV Cache Connector Interface for TensorRT-LLM

Summary

TensorRT-LLM already has several of the control points needed for semantic KV cache reuse. Its KV cache system supports cross-request reuse, prioritized eviction, cache salting, host offload, partial reuse, multimodal UUIDs, and KV cache lifecycle events. Its KV Cache Connector separates scheduler-side orchestration from worker-side KV tensor load/save operations.

This proposal defines a conservative semantic-KV strategy on top of those surfaces:

1. keep TensorRT-LLM exact KV reuse semantics unchanged,
2. add semantic donor discovery as an optional external/provider decision,
3. start with discovery-only telemetry and exact-equivalent materialization,
4. use connector metadata to describe validated donor load plans,
5. use KV cache events to keep donor freshness aligned with engine state,
6. require explicit TensorRT-LLM support before any non-identical semantic KV is materialized.

The first target is safe, prefix-shaped or block-aligned reuse:

| exact local prefix | external exact-equivalent donor blocks | tokens to compute |

With the current public TensorRT-LLM surface, a semantic provider may find a similar donor, but the connector should only report matched tokens or issue a load plan when the target KV can be loaded into TensorRT-LLM-owned blocks without changing exact cache correctness. Non-identical semantic donor KV should remain discovery-only until TensorRT-LLM has an explicit representation for request-local approximate reuse or a backend-approved materialization path.

Motivation

TensorRT-LLM exact KV reuse is valuable when requests share the same prompt prefix or exact reusable blocks. Many long-context enterprise workloads miss that path even though the expensive content is semantically related:

• the same document is queried through different instructions,
• RAG chunks are reordered,
• agent prompts carry repeated tool schemas and context with different surrounding text,
• multi-turn prompts paraphrase or partially restate prior context,
• enterprise users ask related questions over the same policies, tickets, filings, or code.

Semantic donor discovery can expose likely reuse opportunities in these cases and others as the research develops.

This proposal is about engine-side hooks to support semantic KV cache reuse - not about putting a semantic search algorithm inside TensorRT-LLM.

Current SGLang/SemBlend Signal

The SGLang/SemBlend work is our closest engine-local proof point. It shows that a semantic provider can identify donor candidates, hand bounded evidence to an inference runtime, and let the backend either materialize reuse or decline without changing normal exact-cache behavior.

We believe that early results are promising enough to justify defining equivalent engine-native hooks for TensorRT-LLM (and vLLM).

Current TensorRT-LLM Surface

KV Cache System

TensorRT-LLM stores KV state in fixed-token blocks. Filled blocks are placed in a search structure so later requests can reuse matching prefix state. Blocks remain reusable until evicted. Eviction is priority-aware LRU, and host offload can extend the lifetime of reusable blocks by moving them between primary and secondary memory.

Important existing controls for semantic KV integration:

• enable_block_reuse: controls cross-request block reuse.
• cache_salt: isolates reuse to requests with the same salt, useful for tenant or policy boundaries.
• multi_modal_uuids: provides deterministic cache identifiers for multimodal inputs.
• retention policy and block priority: let requests assign priority and duration to token ranges.
• host offload: keeps reusable blocks available after primary-memory pressure.
• partial reuse and copy_on_partial_reuse: relevant to future exact partial-block reuse.
• dtype, attention window, MQA/GQA/VGQA/VSWA layout: part of the cache namespace for safe reuse.

Semantic reuse must respect all of these compatibility boundaries. A provider hit is not valid unless the target request and donor share a safe model, tokenizer, block, dtype, layout, salt, adapter, modality, and connector namespace.

KV Cache Connector

The KV Cache Connector is the most direct extension point for external KV discovery and materialization.

The public connector model separates two roles:

• Scheduler or leader: decides what needs to be loaded or saved, builds connector metadata, and receives request/block information.
• Worker: receives metadata and performs actual KV tensor load/save operations on GPU processes.

Relevant scheduler-side methods:

build_connector_meta(scheduler_output) -> object
get_num_new_matched_tokens(request, num_computed_tokens) -> tuple[int, bool]
request_finished(request, cache_block_ids) -> bool
update_state_after_alloc(request, block_ids) -> None

Relevant worker-side methods:

register_kv_caches(kv_cache_tensor)
start_load_kv(stream)
wait_for_layer_load(layer_idx, stream)
save_kv_layer(layer_idx, stream)
wait_for_save(stream)
get_finished(finished_gen_req_ids, started_loading_req_ids)

This is enough to model a conservative semantic connector:

• scheduler lookup finds a donor,
• scheduler builds opaque donor load metadata,
• allocation binds target block IDs,
• worker loads donor KV into the allocated recipient blocks,
• worker waits before attention consumes each layer,
• request finish saves or registers newly reusable donor blocks.

The proposal should therefore attach to the existing connector lifecycle rather than adding a separate RadixCache-like path. TensorRT-LLM should not need an SGLang-style match_prefix hook. The connector can ask for semantic evidence during get_num_new_matched_tokens, bind destination blocks during update_state_after_alloc, serialize load plans through build_connector_meta, and realize or decline on the worker through start_load_kv and wait_for_layer_load.

KV Cache Events

TensorRT-LLM exposes cache lifecycle events for created, stored, updated, and removed blocks. These events can be consumed to build an eventually consistent view of which blocks exist, where they live, whether they moved between memory levels, and when they are removed.

For semantic KV, these events are the right lifecycle source for donor freshness:

• Stored: a new donor block or span may become discoverable.
• Removed: a donor must be evicted from semantic lookup or quarantined.
• Updated: block priority, memory level, or state may affect materialization cost.
• Created: pool/block allocation can refresh capability and layout metadata.

Events should be treated as freshness evidence, not materialization authority. A connector still validates donor state at load time.

Dynamo TensorRT-LLM Backend

Dynamo already supports TensorRT-LLM as a backend, including KV cache transfer in disaggregated serving, DP-rank routing, and KVBM integration. A TensorRT-LLM semantic connector should compose with that path:

• Dynamo may use semantic donor evidence for placement.
• TensorRT-LLM remains responsible for connector load/save and block lifecycle.
• KVBM/NIXL/UCX transfer semantics remain backend-owned.
• Backend-confirmed materialization is reported separately from semantic placement.

Proposed Integration Modes

Mode 1: Discovery-only

The connector runs semantic lookup and emits telemetry, but it does not report matched tokens and does not load KV.

Use this mode for:

• correctness validation,
• negative controls,
• latency-budget measurement,
• donor freshness validation,
• comparing semantic candidates against exact cache events.

Rules:

• get_num_new_matched_tokens returns (0, False) or equivalent no-load behavior.
• Provider hits are logged as discovery candidates only.
• No KV cache state is changed.
• No ROI is reported as materialized reuse.

Mode 2: Exact-equivalent materialization

The connector may report matched tokens only when the donor KV is exact-equivalent to the target token/hash/cache namespace.

Examples:

• remote/offloaded exact blocks found through semantic donor metadata but verified by token/block identity,
• exact repeated chunks that are block-aligned and namespace-compatible,
• donor token IDs that re-query TensorRT-LLM exact reuse successfully,
• cross-executor exact blocks whose lifecycle is confirmed before load.

Rules:

• exact TensorRT-LLM cache semantics remain unchanged,
• the connector validates token identity and namespace before reporting tokens,
• worker load writes into recipient-owned allocated blocks,
• loaded blocks can only enter exact reuse state if TensorRT-LLM would have accepted them through the normal exact path.

Mode 3: Donor-informed placement

A TensorRT-LLM executor that has useful donor inventory may serve a given request, but TensorRT-LLM may still decline materialization.

This mode is valuable even before non-identical materialization exists because placement can improve exact or high-quality semantic-equivalent reuse probability and keeps donor locality high.

Rules:

• semantic placement is reported separately from backend-confirmed materialized reuse,
• TensorRT-LLM validates donor generation, salt, namespace, and block lifetime,
• backend decline is not an inference failure,
• stale donor and timeout are normal fallback outcomes.

Mode 4: Experimental non-identical semantic materialization

This should remain disabled until TensorRT-LLM exposes an explicit safe representation for approximate or request-local KV reuse.

Required capabilities before enabling:

• prevent approximate donor KV from being inserted as exact reusable cache state,
• bind loaded donor KV to recipient-owned request state,
• support quality gates and negative-control validation,
• report materialized token count and decline reasons,
• handle cancellation, load failure, and eviction without partial commit,
• define behavior for RoPE/position correction or recomputation if required.

Connector Mapping

Request Arrival

On request arrival, TensorRT-LLM exact reuse should run first. If exact overlap is sufficient, semantic lookup is skipped.

If exact overlap is weak, the connector can call an optional provider:

match = await provider.lookup(
    token_ids=request.tokens,
    prompt_text=request.prompt_text_or_none,
    model_id=model_id,
    namespace=cache_namespace,
)

The provider returns None on miss, timeout, policy denial, or unsupported namespace.

A TensorRT-facing match should preserve fields that matter across engines, expressed with TensorRT-native refs:

@dataclass
class SemanticTrtMatch:
    donor_id: str
    reusable_token_count: int
    match_type: Literal["exact_equivalent", "semantic_candidate"]
    donor_token_ids: list[int] | None = None
    donor_block_refs: list["SemanticTrtBlockRef"] | None = None
    segments: list["SemanticTrtSegment"] | None = None
    quality_signals: Mapping[str, Any] | None = None
    provider_generation: str | None = None
    metadata: Mapping[str, Any] | None = None


@dataclass
class SemanticTrtSegment:
    target_start: int
    donor_start: int
    token_count: int
    donor_block_refs: list["SemanticTrtBlockRef"]
    layer_recompute_mask: list[bool] | None = None

In v1, segments are discovery diagnostics unless TensorRT-LLM exposes an execution plan that can avoid computing reused target positions and avoid publishing approximate KV as exact cache state.

get_num_new_matched_tokens

This method is the materialization promise boundary.

It may return a positive token count only when:

• exact-equivalent KV is known to be available, or
• TensorRT-LLM has an explicit safe semantic-materialization mode for the match.

Otherwise, it returns no match and records discovery telemetry.

For asynchronous load, the connector returns (matched_tokens, True) only after it has enough validated donor metadata to complete the load once TensorRT-LLM allocates recipient blocks.

update_state_after_alloc

After TensorRT-LLM allocates blocks, the connector records a load plan:

@dataclass
class SemanticTrtLoadPlan:
    request_id: str
    route_id: str | None
    donor_id: str
    provider_generation: str
    cache_namespace: str
    expected_mode: Literal["exact_equivalent", "semantic_experimental"]
    target_block_ids: list[int]
    donor_block_refs: list["SemanticTrtBlockRef"]
    token_count: int
    semantic_score_bucket: str

Target block IDs are recipient-owned. Donor refs are opaque connector/provider handles until the worker validates and loads them.

This should follow the no-partial-commit invariant we validated in SGLang: allocation happens before request-state mutation. If target blocks cannot be allocated, the connector returns a miss and no donor state is attached to the request. If donor validation fails after allocation, the load plan is discarded and TensorRT-LLM recomputes or falls back through its normal path.

build_connector_meta

The scheduler serializes load plans into connector metadata and broadcasts them to workers.

Metadata must include only what workers need to load safely:

• request ID,
• route ID if supplied upstream,
• donor ID,
• provider generation,
• namespace,
• expected materialization mode,
• target block IDs,
• donor block refs or external store refs,
• token count,
• bounded diagnostics.

Worker start_load_kv and wait_for_layer_load

The worker resolves the donor refs, validates freshness, and copies KV into target blocks on the CUDA stream.

Hard rules:

• unknown generation is stale,
• removed donor refs are stale,
• namespace mismatch is stale,
• unsupported layout is a miss,
• partial load failure invalidates the entire semantic load plan,
• attention must not read a layer until that layer's target KV has been loaded or the request has fallen back to recompute.

If TensorRT-LLM later supports non-identical semantic materialization, the worker-side logic should preserve the same realization invariants SGLang validated while staying TensorRT-native:

• use recipient-owned destination blocks,
• copy from donor refs only after donor refs are protected,
• apply position correction or selective recomputation only through explicit TensorRT-supported primitives,
• clear per-request load handles after one consumption so chunked prefill, retry, or cancellation cannot replay a stale materialization,
• report materialized tokens only after all required layers are available.

request_finished and save_kv_layer

When a request finishes, the connector may register newly reusable donor state.

Registration should include:

• request ID or donor ID,
• model route key,
• cache namespace,
• token count,
• block refs or stored event references,
• provider generation,
• salt or policy scope when safe,
• TTL/freshness metadata,
• optional prompt text only when policy permits embedding.

Embedding and donor index insertion must not run on the TensorRT-LLM scheduler hot path. If prompt text is available, semantic indexing should happen asynchronously and stale registrations must be dropped after generation reset or cache reset.

The SGLang/SemBlend adapter stashes an opaque KV handle synchronously, then offloads embedding and donor-store insertion. We think that TensorRT-LLM should use the same scheduling principle:

1. synchronously record a stable donor ref only after TensorRT-LLM owns the cache blocks,
2. publish or store the engine-native donor metadata needed for future validation,
3. asynchronously embed and index prompt text when policy permits,
4. reject any background registration whose generation no longer matches the engine cache generation.

KV Cache Events

The semantic provider should consume KV cache events when available.

Mapping:

TensorRT-LLM event	Semantic action
Created	Refresh capability, pool, layout, or generation metadata.
Stored	Register or refresh exact donor block refs for future lookup.
Updated	Update donor memory level, priority, or materialization cost.
Removed	Evict or quarantine donor refs immediately.

Events are eventually consistent, so materialization still validates at load time.

Namespace and Compatibility

The semantic cache namespace must include every field that can affect KV correctness:

• model ID and engine version,
• tokenizer identity,
• chat template or prompt format,
• TensorRT-LLM backend mode,
• tokens per block,
• KV cache dtype,
• KV layout and packed-layer format,
• attention window configuration,
• MHA/MQA/GQA/VGQA/VSWA shape,
• LoRA or adapter ID,
• p-tuning extra IDs where applicable,
• speculative decoding mode,
• multimodal UUID/content hash policy,
• cache salt or tenant policy scope,
• connector version,
• provider generation.

If any required namespace component is unknown, semantic materialization is disabled for that request. Discovery-only may still run if policy allows it.

Safety Rules

1. Exact TensorRT-LLM reuse wins before semantic lookup.
2. Semantic search is evidence, not authority.
3. get_num_new_matched_tokens is a promise that valid KV can be loaded.
4. Provider errors, timeouts, cancellation, stale donor, namespace mismatch, and unsupported layout are semantic misses.
5. Approximate donor KV must not be committed as exact reusable cache state.
6. Donor refs must be validated after allocation and before worker load.
7. Donor refs must be revalidated after async wait if load is delayed.
8. Cancellation must release pins and prevent orphaned load plans.
9. Materialized reuse is counted only after worker/backend confirmation.
10. Tenant or salt mismatch fails closed.

Quality and Profitability Gates

For v1, TensorRT-LLM should not implement SemBlend-specific semantic quality policy. It should accept a provider decision only through bounded, generic fields:

• reusable token count,
• exact-equivalence flag,
• match type,
• similarity bucket,
• token overlap bucket,
• namespace compatibility result,
• stale/fresh generation,
• expected materialization mode,
• provider confidence bucket.

The provider can own richer quality policy. TensorRT-LLM should log bounded diagnostics and enforce engine safety.

Profitability should consider:

• matched token count,
• blocks saved,
• current GPU load,
• donor memory level,
• host/offload transfer cost,
• load latency,
• exact cold prefill estimate,
• batch disruption risk.

If estimated saved prefill work does not exceed lookup/load overhead by a deployment-specific margin, the connector should fall back.

Observability

Expose distinct counters and latency histograms:

• semantic lookup total,
• discovery-only hit total,
• exact-equivalent materialized total,
• experimental semantic materialized total,
• backend-declined total,
• stale donor total,
• namespace mismatch total,
• provider timeout total,
• load failure total,
• embedding latency if embedding runs in-process,
• connector metadata build latency,
• load latency by memory level,
• materialized token count,
• TTFT delta for accepted materializations.

Do not collapse these into a single "semantic hit" metric. A semantic placement that the backend declines is useful telemetry, but it is not materialized reuse.

Prototype Plan

Phase A: Discovery-only TensorRT-LLM adapter

• Implement a connector/provider shim that receives request tokens and namespace metadata.
• Run semantic lookup under a hard timeout.
• Return no external matched tokens.
• Emit discovery telemetry and negative-control metrics.
• Consume KV cache events to verify donor freshness behavior.

Acceptance:

• no change to normal TensorRT-LLM exact reuse,
• no materialization claims,
• stale donor and provider timeout are observable,
• benchmark can show where semantic placement would have gone.

Phase B: Exact-equivalent materialization

• Register exact donor refs through KV events or request_finished.
• Validate donor token/block identity and namespace.
• Use get_num_new_matched_tokens only for exact-equivalent spans.
• Build connector metadata with target block IDs and donor refs.
• Load into recipient-owned blocks and report materialized token count.

Acceptance:

• exact repeat still takes normal exact path,
• exact-equivalent external load works,
• stale donor fails closed,
• partial load failure recomputes or falls back without corrupting request state,
• materialized reuse is backend-confirmed.

Phase C: Dynamo TensorRT-LLM route-advice integration

• Let the SemBlend semantic provider supply donor evidence to Dynamo.
• Dynamo validates worker/DP-rank/load/topology eligibility.
• TensorRT-LLM connector validates and materializes or declines.
• Join Dynamo route outcomes with TensorRT-LLM materialization feedback.

Acceptance:

• semantic placement is separate from materialized reuse,
• Dynamo remains the routing authority inside Dynamo deployments,
• TensorRT-LLM remains the materialization authority.

Phase D: Experimental semantic materialization

Only after TensorRT-LLM exposes the required semantic-safe representation:

• request-local approximate KV state,
• no exact-cache pollution,
• explicit quality gates,
• safe cancellation and rollback,
• backend-confirmed reuse accounting.

The SGLang prototype has shown what this mode will need in practice: target-position-aware segment plans, donor lifetime protection, recipient-owned destination slots, position correction, and strict rollback on allocation or copy failure. TensorRT-LLM should not enable this mode until those concepts have TensorRT-native equivalents in the connector/scheduler/worker lifecycle.

Open Questions for TensorRT-LLM Maintainers

1. Can connector-loaded KV be used for the current request without being inserted into exact reusable cache state?
2. Is there a stable way to expose block refs, block hashes, cache salt, memory level, tokens, and generation to an external provider?
3. Should semantic donor refs come from KV cache events, connector registration, or both?
4. Can a connector pin donor blocks or otherwise protect them while a load plan is pending?
5. How should load failure trigger recompute without leaving partially populated blocks visible to attention?
6. Can retention priority be updated from donor value or route feedback without changing exact cache semantics?
7. What is the right compatibility boundary for LoRA, p-tuning, multimodal UUIDs, speculative decoding, VGQA, and VSWA?
8. Should Dynamo TensorRT-LLM integration consume semantic evidence first, or should the first prototype be pure TensorRT-LLM connector discovery?
9. What latency budget is acceptable for a lookup on the scheduler path?
10. What metrics should be added so semantic placement, materialization, and backend decline are distinct?
11. Is there a TensorRT-native equivalent of donor pinning or lock-ref that can protect loaded donor refs during async materialization?
12. Can connector metadata carry a future segmented load plan with target positions without changing scheduler accounting incorrectly?
13. Should TensorRT-LLM expose an explicit "request-local external KV" mode so approximate donor KV cannot become exact reusable cache state?

Tests

Benchmark scenarios:

• exact repeated system prompt,
• partial exact overlap,
• paraphrased long-document QA,
• reordered RAG chunks,
• multi-turn agent prompt with repeated tool schema,
• 8K, 16K, and 32K long-context prompts,
• host-offload donor path,
• high-load batch where lookup overhead must not reduce throughput,
• negative controls.

Report:

• cold TTFT,
• exact warm TTFT,
• discovery-only hit rate,
• semantic placement rate,
• backend-confirmed materialized reuse,
• backend-declined reuse,
• materialized token count,
• lookup latency,
• load latency,
• route overhead,
• ROUGE-L, F1, PPL/logprob, LLM-as-judge quality.

Feedback Period.

2-3 weeks ideally!

CC List.

@PeaBrane
@ishandhanani
@jthomson04
@ibifrost

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.