[DeepSeek-V4] Async CUDA illegal memory access in MTP spec-decode sampler under sustained load with per-size CUDA graphs

Summary

Serving DeepSeek-V4-Flash (sparse-MLA + MTP speculative decoding) via trtllm-serve on the PyTorch backend crashes with an asynchronous CUDA_ERROR_ILLEGAL_ADDRESS (700) under sustained high-concurrency load.

The crash is reproducible on a pristine upstream feat/deepseek_v4 image with zero local patches when CUDA graphs are configured with an exact (dense) per-size capture list (cuda_graph_config.batch_sizes = [1..128], enable_padding: false). Under a sustained synthetic load that pins the scheduler at ~124-128 concurrent requests, the engine hits a device-side assert / illegal memory access in ~4 minutes.

The same configuration with the default sparse / padded CUDA-graph set is stable for 24h+ under identical load. The error is detected (but does not originate) at the MTP spec-decode sampler's CUDA-event synchronize:

py_executor.py _update_requests -> sampler.update_requests
  -> spec_sampler_base.py update_requests -> state.sampler_event.synchronize()
torch.AcceleratorError: CUDA error: an illegal memory access was encountered

Because the error is async, this traceback shows the detection point (the next CUDA sync after the corruption), not the origin. The signature is: MTP spec-decode sampler × exact-per-size CUDA-graph replay at mid-range batch sizes (124-127) → batch-dimension out-of-bounds access.

This is a race, not a deterministic out-of-bounds: setting CUDA_LAUNCH_BLOCKING=1 makes the crash disappear (ran the scheduler at 128 for ~38 min clean), which is why compute-sanitizer / launch-blocking cannot easily catch it.

Environment

• TensorRT-LLM: 1.3.0rc15.post1, base upstream feat/deepseek_v4 @ 80bccf67256177851fc6b63dedc25cd72f490275
• Clean repro image: stock upstream feat/deepseek_v4 container, no local patches (upstream-dsv4-pristine)
• PyTorch: 2.11.0a0+eb65b36914.nv26.02
• CUDA: 13.1 (V13.1.115); Driver: 590.48.01
• GPU: NVIDIA B300 SXM6 (SM103)
• Backend: PyTorch (trtllm-serve)
• Models affected: deepseek-ai/DeepSeek-V4-Flash and deepseek-ai/DeepSeek-V4-Pro (same DeepseekV4ForCausalLM arch: sparse-MLA + MTP)

Model & config

DeepSeek-V4 forces the sparse-MLA KV-cache path, which routes to KV cache manager v2 (use_kv_cache_manager_v2: True, applied as a model default for DeepseekV4ForCausalLM — there is no v1 fallback for this model). MTP speculative decoding is enabled.

extra_llm_api_options for the crashing (dense) configuration:

# CRASHES in ~4 min at sustained concurrency ~128
cuda_graph_config:
  batch_sizes: [1, 2, 3, 4, 5, ..., 126, 127, 128]   # exact graph for every size 1..128
  enable_padding: false

speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 1

kv_cache_config:
  dtype: fp8
  tokens_per_block: 128
  use_kv_cache_manager_v2: true
  enable_block_reuse: true
  host_cache_size: 0

scheduler_config:
  capacity_scheduler_policy: MAX_UTILIZATION

enable_chunked_prefill: true
max_batch_size: 128
max_seq_len: 131072
disable_overlap_scheduler: true

extra_llm_api_options for the stable (sparse/padded) control — identical except CUDA graphs:

# STABLE 24h+ under identical load
cuda_graph_config:
  enable_padding: true        # ~19 padded graph sizes; no exact graph at 124-127
# ...everything else identical...

The discriminator is purely the CUDA-graph capture set. The warmup log reports it directly:
Creating CUDA graph instances for N batch sizes — N = 128 (dense, crashes) vs ~19 (padded, stable) vs ~34 (default sparse, stable).

Reproduction (pristine upstream, no local patches)

1. Launch trtllm-serve with the stock upstream feat/deepseek_v4 image (no patches) on DeepSeek-V4-Flash, using the dense extra_llm_api_options above (cuda_graph_config.batch_sizes: [1..128], enable_padding: false, MTP=1, fp8 KV, KV-cache-manager-v2, MAX_UTILIZATION).
2. Confirm from the warmup log that 128 CUDA-graph instances were captured (Creating CUDA graph instances for 128 batch sizes).
3. Drive sustained synthetic load that pins the scheduler at ~124-128 concurrent requests (e.g. concurrency-128 completion blast; on a pristine image use sanitized request bodies — raw prod requests with top_k=-1 etc. get 400'd and never load the engine).
4. Result: device-side assert / illegal memory access in ~4 min, surfacing at the MTP spec-decode sampler's sampler_event.synchronize().

Control: Repeat with cuda_graph_config.enable_padding: true (or the default sparse batch_sizes) and everything else identical. The engine is stable for 24h+ under the same load. The default sparse CUDA-graph set has no exact graph for 124-127, so those mid-range batch sizes run eager and never hit the OOB.

Observed crash (verbatim)

Detection at the MTP spec-decode sampler event sync:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 4171, in _update_requests
    self.sampler.update_requests(sample_state, resource_manager)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/speculative/spec_sampler_base.py", line 203, in update_requests
    state.sampler_event.synchronize()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/sampler.py", line 164, in synchronize
    self.cuda_event.synchronize()
  File "/usr/local/lib/python3.12/dist-packages/torch/cuda/streams.py", line 245, in synchronize
    super().synchronize()
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[TRT-LLM] [E] [_torch  ] Encountered an error in sampling: CUDA error: an illegal memory access was encountered
[TRT-LLM] [E] [_torch  ] Fatal error detected, initiating shutdown: CUDA error: an illegal memory access was encountered
[TRT-LLM] [E] [_torch  ] Error in event loop: CUDA driver error: <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> (b'an illegal memory access was encountered')

Once the CUDA context is corrupted, the subsequent request teardown fails inside KV-cache-manager-v2 (a downstream symptom, same underlying error):

Traceback (most recent call last):
  ...
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 4309, in _do_terminate_request
    self.resource_manager.free_resources(request)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/resource_manager.py", line 3765, in free_resources
    kv_cache.close()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/runtime/kv_cache_manager_v2/_core/_kv_cache.py", line 529, in close
    with self._record_event():
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/runtime/kv_cache_manager_v2/_core/_kv_cache.py", line 1787, in _record_event
    self._finish_event = CachedCudaEvent(self.cuda_stream)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/runtime/kv_cache_manager_v2/_utils.py", line 802, in _record
    _unwrap(drv.cuEventRecord(ev, stream))
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/runtime/kv_cache_manager_v2/_utils.py", line 78, in _unwrap
    raise CuError(ret)
tensorrt_llm.runtime.kv_cache_manager_v2._exceptions.CuError: CUDA driver error: <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> (b'an illegal memory access was encountered')

When caught earlier by a device-side assert (instead of surfacing as a raw IMA), it is a PyTorch IndexKernel.cu:111 Assertion '-sizes[i] <= index && index < sizes[i] out-of-bounds, with the offending threads spanning 0-127 — i.e. the **batch dimension** (max_batch_size = 128`). This is the same out-of-bounds access; whether it surfaces as an assert or a raw IMA depends on timing.

Diagnosis

• It is a race / stream-ordering hazard, not a deterministic OOB. CUDA_LAUNCH_BLOCKING=1 makes the crash disappear entirely (scheduler pinned at 128 for ~38 min, clean). Serializing kernel launches closes the race window, so the Python traceback only ever shows the async detection point, and compute-sanitizer / launch-blocking cannot readily catch it.
• Dense vs sparse CUDA-graph control is decisive. Single-variable A/B: the only change between the crashing (~4 min) and stable (24h+) runs is the CUDA-graph capture set. Dense [1..128] captures exact-size graphs for the mid-range batch sizes the scheduler actually pins (124-127); replaying those exact-size graphs with the MTP spec-sampler triggers a batch-dimension out-of-bounds. The default sparse / padded set has no graph at 124-127 (those run eager or get rounded up to a captured size), so the OOB is never hit.
• The detection sites are async artifacts (sampler_event.synchronize() in the spec sampler; KV-v2 event record/scrub during slot release) — the corruption happens earlier in the MTP spec-decode + sparse-MLA forward/decode path while a captured CUDA graph is replayed.

Mitigations tried

These reduce frequency / extend MTTF but do not fully eliminate the residual crash — they point at the affected area (cross-stream tensor lifetime in the MTP-sampler / sparse-MLA / KV-v2 path):

Mitigation	Effect
cuda_graph_config.enable_padding: true (sparse graphs)	Biggest single mitigation — eliminates the dense-graph crash; this is the clean upstream control that runs 24h+.
Upstream a05ccbe4c6 — "DSv4 MLA overlap: record_stream cross-stream tensors"	record_stream on MLA pre_aux / q / topk_indices cross-stream tensors so the caching allocator can't recycle them mid-use. Extends MTTF substantially (~155x in one A/B). Indicates the bug class is cross-stream tensor lifetime / use-after-free.
Upstream d15f21c8d3 — "Fix V2 context finalization ordering"	Moves KV-v2 context finalization to run before request termination (freed KV was being accessed after finalize). Extends MTTF further.

Even with all of the above, a residual crash with the same signature remains: MTTF grows from minutes (heavy synthetic concurrency-128) to hours (~21h observed under light real load), but it still recurs. So there is at least one more cross-stream / lifetime race in the MTP spec-sampler / sparse KV-cache-v2 path.

Additional notes / caveats

• The clean, attributable upstream finding is the dense-CUDA-graph IMA, reproduced on the pristine upstream feat/deepseek_v4 image with zero local patches. The single-variable dense-vs-sparse control isolates the CUDA-graph capture set as the cause. This issue is framed primarily around that case.
• The residual race (post-padding, post-a05ccbe4c6, post-d15f21c8d3) was observed on a build that additionally carries local patches (an MTP-logprobs CUDA kernel and KV-manager C++ changes), so the residual race is not cleanly attributable to upstream. We mention it because it has the identical signature (sampler_event.synchronize() → CUDA_ERROR_ILLEGAL_ADDRESS → KV-v2 CuError) and the same async / CUDA_LAUNCH_BLOCKING-hides-it behavior, strongly suggesting a remaining upstream cross-stream hazard in the MTP-sampler / KV-v2 path that the dense graphs merely surface earlier.
• Both DeepSeek-V4-Flash and DeepSeek-V4-Pro are affected (same DeepseekV4ForCausalLM arch, sparse-MLA + MTP).

Ask

1. Confirm the batch-dimension out-of-bounds when exact-size CUDA graphs are captured/replayed at mid-range batch sizes with the MTP spec-decode sampler (dense cuda_graph_config.batch_sizes), and fix it so dense graphs are safe with MTP.
2. Investigate the residual cross-stream / use-after-free race in the MTP spec-sampler / sparse-MLA / KV-cache-v2 path that persists even with sparse graphs + a05ccbe4c6 + d15f21c8d3 (same async IMA signature; hidden by CUDA_LAUNCH_BLOCKING=1).

Reproduce the dense-versus-sparse CUDA-graph comparison first, then trace the reported path through py_executor.py, spec_sampler_base.py, sampler.py, and the KV-cache-manager-v2 files named in the traceback. Confirm where the batch-dimension access or cross-stream lifetime failure originates during MTP spec-decode replay. Done means dense per-size graphs remain stable under the stated sustained-load scenario without the illegal-memory-access signature.