[Bug]: KVCacheManagerV2 raises 'History length cannot exceed capacity' in update_resources on a PP stage-2 rank (DSA model, block reuse off)

System Info

• GPU: 8x NVIDIA H200 NVL (SM90; two 4-GPU NVLink islands, cross-island over PCIe/UPI), driver 595.71.05
• Host: Ubuntu 24.04.4, 2 TiB RAM, Xeon 6747P
• Container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc25 (CUDA 13.2, Python 3.12.3, torch 2.12.0a0+nv26.5, transformers 5.5.4)
• TensorRT-LLM: 1.3.0rc25 (tag v1.3.0rc25, 785c948); the code paths cited below were also checked on main @ a6616d6f (2026-09-03)
• Backend: PyTorch (trtllm-serve --backend pytorch)

Who can help?

@yizhang-nv (KV cache manager V2 for DSA, #16060)

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Same model, image and traffic as #18660 (GLM-5.3 W4AFP8 on the DeepSeek-V3.2 path, pp_size 2 --tp_size 4 --ep_size 4, MTP draft 1, KV cache manager V2, max_seq_len 262144, enable_chunked_prefill: true, max_num_tokens 8192, 16 requests in flight, prompts ~146k tokens), but with kv_cache_config.enable_block_reuse: false and --max_batch_size 16. After 115 requests the stage-2 rank (rank 4) raised in update_resources:

[TRT-LLM] [E] [_torch][RANK 4] Error in event loop: History length cannot exceed capacity
  File ".../tensorrt_llm/_torch/pyexecutor/py_executor.py", line 3246, in _handle_executed_batch
    self.resource_manager.update_resources(
  File ".../tensorrt_llm/_torch/pyexecutor/resource_manager.py", line 3018, in update_resources
    resource_manager.update_resources(scheduled_batch,
  File ".../tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py", line 3989, in update_resources
    success = kv_cache.resize(new_capacity, history_length)
ValueError: History length cannot exceed capacity
[TRT-LLM] [E] [_torch][RANK 7] Executor loop crashed on this rank; hard-killing all 8 ranks in 9.99947s (peers cannot make progress without this rank). Set TLLM_RANK_CRASH_HARD_KILL_GRACE=-1 to disable.

The request that triggered it was a long chunked-prefill context (146k-169k prompt tokens, max_completion_tokens 16384, non-streaming). Only rank 4 (first rank of pipeline stage 2, which also hosts the MTP layer) raised.

Expected behavior

resize(new_capacity, history_length) is called with a capacity that covers the history, or the executor pauses/rejects the request; either way the engine keeps serving.

actual behavior

ValueError in the executor loop on one rank, followed by the rank-crash hard kill of all ranks; the HTTP frontend stays alive answering 503 (#18663).

additional notes

• Not the SWA min-slot underflow of #15401 / #15462: this model has no sliding window and the failure is in update_resources, not in capacity planning.
• The same traffic on the V1 manager (use_kv_cache_manager_v2: false) ran 245 requests before a different failure (HangDetector, no Python traceback), so V1 is not a clean workaround either (and V1 loses MTP acceptance, #18662).
• Happy to run a candidate fix on the same replay.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start in tensorrt_llm/_torch/pyexecutor/py_executor.py at _handle_executed_batch, then trace update_resources through resource_manager.py and kv_cache_manager_v2.py to the resize call around line 3989. Reproduce the PP stage-2, long chunked-prefill scenario with block reuse disabled and inspect the capacity and history values. Done means the engine avoids the ValueError by covering the history or safely pausing or rejecting the request.