[Bug][Disagg] KvCacheAwareRouter silently degrades to load-only routing (per-server block table not populated; matched_tokens≡0) — ~90% vs 96% prefix-cache hit

Summary

On feat/deepseek_v4, multi-turn disaggregated serving with the orchestrator router type: kv_cache_aware reaches only ~90% prefix-cache hit, versus ~96% for the conversation router under identical KV-cache settings (same model, parallelism, host-cache, dataset, concurrency — only the router differs).

Evidence indicates the orchestrator's per-server KV block table is never populated in this disagg path, so KvCacheAwareRouter.get_next_server() computes matched_tokens == 0 for every server and silently falls back to pure least-loaded routing (no warning/error). As a result ~8–9% of follow-up turns are routed to a CTX server that does not hold the conversation's prefix and get re-prefilled.

Setting router_args.backfill_block_hashes_on_finish: true — which injects each served request's block hashes into the router's server-state on finish, bypassing the /kv_cache_events path — recovers cache hit to 95%. tokens_per_block has no effect, consistent with an empty block table.

Environment

• Branch feat/deepseek_v4 (container built ~f1723105a1; current HEAD 41a16de). PyTorch backend.
• Model DeepSeek-V4-Pro, disaggregated 3× CTX (DEP4: TP4/EP4, attention-DP) + 1× GEN (DEP8), MTP=3, FP8 KV, CTX host_cache_size 120 GiB, enable_block_reuse: true, c=384.
• Workload: multi-turn agentic-coding trajectories (avg ISL ≈ 40k tokens, avg OSL ≈ 300), closed-loop; a per-trajectory id is sent as X-Correlation-ID (used by the conversation router; ignored by kv_cache_aware).

Mechanism — tensorrt_llm/serve/router.py

KvCacheAwareRouter.get_next_server() scores each candidate CTX server (≈ L1015–1032):

score = matches[-1] / padded_tokens - workloads[i] / self._max_batch_size   # L1025

• matches[-1] = ServerState.matched_tokens(block_hashes, hash_algo) (L205–219), which counts the request's prefix block-hashes present in that server's block_table, breaking on the first miss.
• block_table is populated only by ServerState.update_with_events() (L180–197) from polling each server's /kv_cache_events endpoint (poll_events L199–203; poll_and_update L227–238 logs only on failure).
• workloads[i] = num_active_requests; self._max_batch_size default 64 (L949).

If the block_table is never populated, then matched_tokens == 0 for all servers → score = − workloads[i] / max_batch_size → pure least-loaded selection, silently (no log indicating the cache signal is dead).

Evidence

All runs are the same config except the router knob. Cache hit = server-reported cached_tokens / input_tokens over the measurement window. "Turn scatter" = fraction of follow-up turns (turn ≥ 2 of a trajectory) whose server_cached_tokens < 50% of server_input_tokens despite the prefix having been served on a prior turn (computed from per-request logs).

Key observations:

1. tokens_per_block is a no-op. The orchestrator defaults to tokens_per_block=32 while the CTX servers store 128-token KV blocks (kv_cache_hash_algo: v1_block_key). Aligning the orchestrator to 128 changed nothing (89.6% → 89.8%, scatter 8.6% → 8.6%). The orchestrator log confirms the setting took effect: BlockHashMixin: tokens_per_block=32 → ...=128. A hash-granularity mismatch cannot explain the gap if there is nothing in the block table to match against — which this no-op is consistent with.
2. No /kv_cache_events poll failures were logged ("Failed to poll KV cache events" count = 0), yet the cache-match signal is evidently inert — i.e. the table is simply not being populated (rather than polling erroring out).
3. backfill_block_hashes_on_finish: true recovers the hit (L1037–1067: stash block hashes at routing time, add_blocks into server-state on finish — independent of /kv_cache_events). Alone → 92.8%; combined with max_batch_size: 512 (shrinks the load term so the now-non-zero cache term dominates) → 95.0%, scatter 3.2% — approaching the conversation router.

Questions / requests for maintainers

1. Is /kv_cache_events-driven block_table population expected to work for KvCacheAwareRouter in disaggregated + attention-DP serving, or is backfill_block_hashes_on_finish the intended mechanism there? (The naming suggests backfill is a disagg workaround — if so it isn't documented as required.)
2. If the events path is supposed to populate the table here, this looks like a bug: the CTX servers don't appear to surface their stored blocks to the router, so kv_cache_aware silently degrades to a load balancer with no diagnostic. A warning when a server's block_table stays empty under nonzero traffic would make this visible.
3. The score load term workloads[i] / max_batch_size (default 64) easily dominates the [0,1] cache term at high concurrency; the cache-affinity benefit only materialized once we raised max_batch_size to 512. Consider documenting this interaction / a more scale-robust normalization.

Minimal repro (router_args)

context_servers:
  router:
    type: kv_cache_aware
    use_tokens: false
    # default -> ~90% prefix-cache hit on multi-turn disagg, ~8.6% turn scatter
    # vs. adding:
    backfill_block_hashes_on_finish: true
    max_batch_size: 512
    # -> ~95% prefix-cache hit, ~3.2% scatter

Caveat on methodology

We did not directly instrument matched_tokens (the orchestrator does not log the per-server match values). The matched_tokens ≡ 0 conclusion is inferred from (a) tokens_per_block being a no-op and (b) backfill — which bypasses the events path — recovering the hit. A one-line log of matches/scores inside get_next_server() would confirm directly, and we're happy to run that if useful.

Start in tensorrt_llm/serve/router.py, especially KvCacheAwareRouter.get_next_server(), ServerState.update_with_events(), poll_events(), and poll_and_update(). Reproduce the disaggregated kv_cache_aware configuration, then inspect whether /kv_cache_events populates each server's block_table and whether matched_tokens remains zero. Done means the cache-aware signal works or its unsupported path is diagnosed, with a regression test or diagnostic behavior covering the failure.