[Bug]: KVCacheManagerV2 + DSA: indexer (draft) cache pool is not part of admission; exhaustion under prefix-shared long-context load aborts the engine

System Info

• GPU: 8x NVIDIA H200 NVL (SM90; two 4-GPU NVLink islands, cross-island over PCIe/UPI), driver 595.71.05
• Host: Ubuntu 24.04.4, 2 TiB RAM, Xeon 6747P
• Container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc25 (CUDA 13.2, Python 3.12.3, torch 2.12.0a0+nv26.5, transformers 5.5.4)
• TensorRT-LLM: 1.3.0rc25 (tag v1.3.0rc25, 785c948); the code paths cited below were also checked on main @ a6616d6f (2026-09-03)
• Backend: PyTorch (trtllm-serve --backend pytorch)

Who can help?

@yizhang-nv (KV cache manager V2 for DSA, #16060 / #18093)

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Model: GLM-5.3 (GlmMoeDsaForCausalLM -> DeepSeek-V3.2 path, DSA sparse attention), W4AFP8 checkpoint, pp_size 2 --tp_size 4 --ep_size 4, speculative_config: {decoding_type: MTP, max_draft_len: 1}, kv_cache_config: {enable_block_reuse: true, free_gpu_memory_fraction: 0.8, dtype: auto} (KV cache manager V2 auto-selected: Resolved use_kv_cache_manager_v2='auto' to True for GlmMoeDsaForCausalLM), enable_chunked_prefill: true, max_num_tokens 8192, max_seq_len 262144.

Traffic: replay of real agentic conversations at their original arrival pace, 16-32 requests in flight, prompt length p50 ~146k / p95 ~169k tokens, with long shared prefixes between turns of the same conversation (multi-turn tool use). Startup log:

Splitting KV cache max_gpu_total_bytes: total=60.51 GiB, target=... (49968 bytes/token), draft=... (1284 bytes/token)
KVCacheV2Scheduler: tokens_per_block=64, max_num_tokens=8192, max_batch_size=64, draft_mgr=DSACacheManagerV2, ...

After ~180 requests (max_batch_size 64):

[TRT-LLM] [E] [executor][RANK 0] submit request failed: Draft KV cache context resize failed for request 186: could not resize to 102991 tokens
  File ".../tensorrt_llm/_torch/pyexecutor/py_executor.py", line 2686, in _executor_loop_pp
    self.resource_manager.prepare_resources(scheduled_batch)
  File ".../tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py", line 2759, in _prepare_draft_resources
    raise RuntimeError(
RuntimeError: Draft KV cache context resize failed for request 186: could not resize to 102991 tokens
[TRT-LLM] [E] [_torch][RANK 3] HangDetector: propagating hard-kill to all ranks via MPI_Abort.

Same traffic with max_batch_size 8, after 26 requests:

[TRT-LLM] [E] [_torch][RANK 0] Error in event loop: Failed to resume draft KV cache for request 78
RuntimeError: Failed to resume draft KV cache for request 78
[TRT-LLM] [E] [executor][RANK 0] Broadcasting event-loop error to 16 pending request(s): Failed to resume draft KV cache for request 78

Both go away with kv_cache_config.enable_block_reuse: false (which is what the GLM-5 deployment guide ships), at the cost of losing prefix caching entirely.

Expected behavior

Admission accounts for the DSA indexer ("draft") cache so a context request is only scheduled when both pools can hold it, or the indexer mirror reuses blocks like the primary pool; and a failed resize/resume pauses or rejects the request instead of taking the engine down.

actual behavior

The docstring of _prepare_draft_resources says "The main V2 scheduler only manages the primary KV cache manager", and the mirror calls stop_committing() (no block reuse). Both pools are sized to the same token count, but with block reuse the primary pool holds far fewer blocks than the sum of in-flight prompt lengths on prefix-shared traffic, so admission proceeds while the indexer pool is already full; the resize/resume failure is then fatal for the whole engine (HangDetector MPI_Abort). The HTTP frontend keeps running and answers /health with 503 while every request fails (#18663).

additional notes

• #18093 (one-model draft KV reuse in cache manager V2) looks related because it coordinates target/draft reuse, but the pool that fills here is the DSA indexer K-cache mirror (DSACacheManagerV2, 1,284 B/token), not the speculative draft-model cache; please deduplicate if you consider it the same root cause.
• Reproducible without agentic traffic by sending many long prompts that share a long prefix (so block reuse keeps the primary pool small) at 16+ concurrency.
• Related but different: #15401 / #15462 concern windowed (SWA) pools; this model has no sliding window.
• Sibling reports from the same runs: #18661 (V2 History length cannot exceed capacity with reuse off), #18662 (MTP acceptance with V1), #18658 (indexer wk scale), #18659 (MIXED_PRECISION kv_b_proj).

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.