DeepSeek-V4 + MTP + attention DP: `CUDA_ERROR_ILLEGAL_ADDRESS` in `CUDAGraph::replay()` during executor init

Component: trtllm-bench throughput / _torch runtime / kv_cache_manager_v2
Version: TensorRT-LLM 1.3.0rc21
Hardware: 4x NVIDIA GB300 (284 GB each), driver 580.159.04, CUDA 13.2
Model: DeepSeek-V4-Flash (FP8, model_type: deepseek_v4, num_nextn_predict_layers: 1)
Severity: High — hard crash + segfault during initialization
Likely related: the _num_tables AttributeError in #17024. Both are
triggered by MTP on DeepSeek-V4; this one may be a downstream symptom of the same
broken draft-KV bookkeeping, surfacing as memory corruption instead of a Python
error once CUDA graphs are in play. Worth triaging together.

Summary

With MTP and attention DP both enabled, DeepSeek-V4 crashes during executor
initialization with an illegal memory access inside CUDA graph replay, followed
by a segfault. All ranks fail simultaneously. No benchmark requests are ever
issued.

Reproduction

Config is taken verbatim from examples/models/core/deepseek_v4/README.md
("Benchmarking" section):

enable_attention_dp: true
attention_dp_config:
  batching_wait_iters: 0
  enable_balance: true
  timeout_iters: 60
kv_cache_config:
  tokens_per_block: 128
  dtype: fp8
  free_gpu_memory_fraction: 0.9
cuda_graph_config:
  enable_padding: true
moe_config:
  backend: TRTLLM
speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 1

trtllm-bench --model $M --model_path $M prepare-dataset \
  --output /tmp/dsv4_1k1k.txt \
  token-norm-dist --input-mean 1024 --output-mean 1024 \
  --input-stdev 0 --output-stdev 0 --num-requests 4096

trtllm-bench --model $M --model_path $M throughput \
  --tp 4 --ep 4 --dataset /tmp/dsv4_1k1k.txt \
  --max_batch_size 256 --max_num_tokens 8192 \
  --concurrency 32 --num_requests 256 \
  --kv_cache_free_gpu_mem_fraction 0.9 --config /tmp/bench_config.yml

Crash occurs ~5 min in, immediately after the autotuner phase.

Observed failure

[_torch][RANK 0] Running autotuner warmup...
[_torch][RANK 0] [Autotuner] Autotuning process starts ...
...
[executor][RANK 3] Failed to initialize executor on rank 3:
    CUDA driver error: <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>
    (b'an illegal memory access was encountered')
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
tensorrt_llm.runtime.kv_cache_manager_v2._exceptions.CuError:
    CUDA driver error: <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>
[executor] Executor worker died during initialization.
!!!!!!! Segfault encountered !!!!!!!
  File "<unknown>", line 0, in cuGraphLaunch
  File "<unknown>", line 0, in cudaGraphLaunch
  File "<unknown>", line 0, in at::cuda::CUDAGraph::replay()

All 4 ranks report the same error.

Note for triage: the log then fills with
ValueError: Dereferencing a dangling rawref from
kv_cache_manager_v2/_page.py:454 (_SharedPageLock.__del__ → unlock →
unwrap_rawref). These are teardown noise after the fault, not the cause —
they are by far the loudest thing in the log and will mislead if taken at face
value.

Possibly relevant warning

Emitted shortly before the crash:

[_torch][RANK 0] Skipping general warmup with 256 tokens (256 generation):
    not enough KV cache space on any TP rank.

With enable_attention_dp: true, --max_batch_size 256 is per rank. Unclear
whether the KV shortfall is related, or whether warmup should proceed to graph
capture at all after skipping that step.

Bisect

All runs tp4/ep4, concurrency 32, 128 requests, same model and build:

MTP	attention DP	result
on	on	crash — this issue
off	on	OK — 2382 tok/s output
on	off	crash — AttributeError: '_num_tables' (#17024)
off	off	OK — 3129 tok/s output

MTP is necessary for the failure; attention DP determines which failure.
Neither non-MTP configuration crashes, so this is not a general
model/hardware/build problem.

CAVEAT — deviation from the documented recipe

The README's benchmarking example specifies --tp 8 --ep 8 on an 8-GPU node.
This report used --tp 4 --ep 4 on 4 GPUs, with --concurrency and
--num_requests scaled down; the YAML is otherwise verbatim from the README.

I could not test the documented tp8/ep8 configuration (only 4 GPUs available), so
I cannot claim the README recipe is broken as written — it is possible this is
specific to 4-GPU scaling. Anyone triaging should reproduce at tp8/ep8 before
concluding otherwise. The companion issue #17024 is independent of this
caveat, since it reproduces without attention DP at all.

Context

Configurations that ran successfully on the same machine, model and build:

• trtllm-eval aime25, tp4/ep4, max_batch_size 16, plain config — 30 requests, ~5m20s.
• LLM.generate direct, tp4/ep4, max_seq_len 135168 — 3 requests up to 131k output tokens.
• trtllm-bench throughput, both non-MTP variants above.

None enabled MTP.

Start by reproducing the documented DeepSeek-V4 benchmark from examples/models/core/deepseek_v4/README.md at tp8/ep8, then compare it with the reported tp4/ep4 matrix. Inspect MTP, attention-DP, kv_cache_manager_v2, and CUDA-graph initialization, using #17024 for related behavior; treat _page.py:454 dangling-rawref messages as teardown noise. Done means the supported configuration initializes without the illegal access or segfault.