[RFC]: Reduce overhead in KV cache event publishing

Motivation

Goal: take KV cache event publishing off the scheduler hot path without
changing the event schema.

KV cache events let external consumers such as Dynamo's KV-aware router track
block reuse. Two overheads sit on the publish path for KVCacheManagerV2 with
attention-DP.

1. The block hash is computed twice

The V2 radix tree already computes a 256-bit SHA-256 key per block for prefix
reuse (_block_radix_tree.py:335-341). The default event hash is V1
(kv_cache_hash.py:23), so the event path hashes the same tokens a second time
with the 64-bit legacy mixer to match the C++ event schema
(_event_manager.py:583-632, kv_cache_hash.py:56-71).

The second hash runs on the scheduler hot path, once per block, over tokens the
tree already hashed. A single-hash path exists but is not the default:
kv_cache_event_hash_algo="v2_sha256_64" reuses the radix key
(_event_manager.py:480-485). The C++ V1 manager already hashes once and reuses
block->getHash() in the event (kvCacheEventManager.cpp:107).

2. Events are funneled to rank 0

With attention-DP, every rank runs an event manager. A background thread gathers
all ranks' events onto rank 0 by serialize, send, recv, deserialize
(kvCacheEventManager.cpp:186-251), behind an allreduce barrier every
attention_dp_events_gather_period_ms (default 5 ms). This exists only because
the events API is served on rank 0 (rpc_worker.py:142-164).

The per-rank events are unique, so the data is needed. The gather is not. It
adds an all-rank barrier every 5 ms and makes rank 0 a serialize/deserialize
bottleneck that grows with DP width. Pure TP/PP has no gather because only rank
0 emits, so this applies to attention-DP only.

Measured impact

Internal benchmarking:

• Enabling legacy V1 publication caused a significant throughput regression with
  cache reuse unchanged. The cost is event production, not routing.
• Emitting the low 64 bits of the existing radix key, plus coalescing lifecycle
  events before publish, brought throughput back to near baseline.
• Per-rank publishing, with no rank-0 gather, gave a further improvement.

Proposed Change

Two changes under one RFC. Proposal A carries the larger measured win; Proposal
B is the transport it builds on. The reference drafts (see Draft implementations)
stack B as the base and A on top.

A. Emit the radix key, drop the V1 re-hash

• Change the auto resolution. In get_effective_kv_cache_event_hash_algo()
  (kv_cache_hash.py:38-41), make auto resolve to v2_sha256_64 when
  use_kv_cache_manager_v2 is true. Today it ignores that argument and always
  returns v1_block_key. The user-facing field is
  KvCacheConfig.kv_cache_event_hash_algo (llm_args.py:3522).
• Target v2_sha256_64 specifically because it is the low-64-bits-of-SHA-256
  form the KV-aware router already expects, so producer and consumer match. With
  it, events reuse the SHA-256 digest the tree already computed
  (_event_manager.py:480-485). No second hash.
• Keep v1_block_key selectable for mixed C++/V2 fleets and existing consumers.
• Advertise the algorithm in-band. V2 events already carry hash_algo
  (_event_manager.py:125,428, serialized at _utils.py:1045-1047), and the
  router already keeps per-algo block tables (router.py:154-177). Remaining
  work: have the consumer select its hasher from the advertised hash_algo
  instead of assuming V1. This is a dependency for flipping the default.
• Coalesce lifecycle events before publish. Drop blocks that never survive to
  reuse so they generate no event traffic. It ships with the hash change because
  internal benchmarking measured them together as one fast path, but it is a
  separable concern and can split into its own PR if reviewers prefer.

B. Publish per rank, drop the rank-0 gather

• Each rank publishes its own events, tagged with attentionDpRank
  (executor.h:1846). This removes the gather, the 5 ms barrier, and the rank-0
  bottleneck.
• Needs an event egress on all ranks, not just rank-0's RPC server
  (rpc_worker.py:142-164), and a consumer that subscribes to N streams.
• Validated end-to-end in internal benchmarking: one ZMQ publisher/subscriber
  per rank, all ranks used, cache-read ratio held. This is upstreaming, not new
  research.
• Fallback if deferred: make the gather non-blocking (Isend/Irecv,
  pull-driven, shutdown-only barrier). Keeps the consumer contract and cuts the
  steady-state cost.

Draft implementations

Two draft PRs, used to measure the impact above. Proof of concept, not the
proposed final form. They stack: the native publishing path is the base and the
production optimization builds on it.

• NVIDIA/TensorRT-LLM#16869 — [feat] publish native V2 KV cache events (base).
  Adds a strict KVEventsConfig, replaces the attention-DP gather callback with
  local per-rank event conversion, and publishes vLLM-compatible msgpack over ZMQ
  with base_port + rank endpoints and no KV-event object collectives. Supports
  attention-DP and TP; rejects PP/CP. Paired Dynamo consumer:
  ai-dynamo/dynamo#12167. Covers Proposal B.
• NVIDIA/TensorRT-LLM#16876 — [perf] optimize native V2 KV event production
  (stacked on #16869). Reuses each radix block's existing SHA-256 Block.key
  (low 64 bits to the vLLM wire int) instead of recomputing the V1 hash, and uses
  a scheduler-local manager that emits only publishable full-block store/remove
  events. The PR reports ~98% of constructed events are filtered before the wire.
  Covers Proposal A.

Alternatives Considered

• Raise attention_dp_events_gather_period_ms. Reduces gather frequency but
  keeps the rank-0 fan-in and the barrier, and trades event latency for less
  overhead. Mitigation, not a fix.
• Cache the V1 hash harder instead of dropping it. The path already caches
  per block key (_v1_hash_by_block_key). Caching does not remove the
  first-touch cost, which is one full token hash per new block on the hot path,
  on top of the SHA-256 the tree already did. Does not address the root cause.
• Switch the radix tree key to the 64-bit hash so lookup and event share one
  hash, like C++. Weakens collision resistance. V2 chose 256-bit keys on
  purpose. Rejected.

Testing and Success Criteria

Tests:

• Unit: emitted event hash equals the consumer-recomputed hash for each algo
  (v1_block_key, v2_sha256_64), covering text tokens and the non-text
  fallback path.
• Unit: consumer selects the hasher from the advertised hash_algo; a mismatch
  is caught.
• Integration: with events on, V2, attention-DP, the per-rank publish path
  produces the same block set as the current gather path.
• Benchmark: reproduce publish-on vs publish-off throughput on the V2 path.

Success criteria:

• Publish-on throughput close to publish-off baseline.
• No cache-reuse regression.
• Hash-parity tests green for every supported algo.

Compatibility and Migration

• No change to the public KVCacheEvent schema.
• C++ V1 manager is unchanged; it already single-hashes.
• Default change is scoped to V2 auto resolution; v1_block_key stays
  selectable.
• Sequencing: flip the default only after consumers select their hasher from the
  advertised hash_algo.

Open Questions

• V1 deprecation window. During a rolling upgrade a fleet runs mixed C++ (V1)
  and V2 engines. If V2 auto flips to v2_sha256_64, co-existing producers
  emit different hashes for the same block. How long do we keep V1 the default,
  or do we gate the flip behind one release?
• Per-rank publishing: the draft (#16869) uses one ZMQ publisher/subscriber per
  rank at base_port + rank, with the paired Dynamo consumer in
  ai-dynamo/dynamo#12167. Is that the endpoint model to standardize, and do
  cross-rank parent_hash references still resolve when streams arrive
  independently?
• Does per-rank publishing change the non-attention-DP path, where only rank 0
  emits today, or does that path stay as-is?
• Topology support. The native publishing draft supports attention-DP and TP and
  rejects PP/CP. Extending to PP/CP is open.

Feedback Period

~1 week.

CC List

• @NVIDIA/trt-llm-kv-cache-manager-devs: kvCache*, blockKey*,
  kv_cache_manager_v2, resource_manager.py
• @NVIDIA/trt-llm-runtime-devs: batch_manager, executor
• Add the Dynamo KV-events integration reviewers.

Any Other Things

• The rank-0 mechanism is a point-to-point gather behind an allreduce barrier,
  not MPI_Allgather. Pure TP/PP has no gather. Stated here so the term is not
  misread.
• Scope is the publish path and transport only. It does not cover KV-routing
  policy or the open prefill/decode routing-interaction gap.