[Bug] mhc_pre "CUDA error: invalid configuration argument" on MI300X (gfx942) during DeepSeek-V4-Flash-0731 profile_run
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 565
- Forks
- 585
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 366
Description
Environment
- GPU: 8x AMD Instinct MI300X (gfx942)
- ROCm: 7.1.1, PyTorch
2.14.0.dev20260803+rocm7.1 - aiter: reproduced on both
v0.1.19andmain@17f24ec6e(2026-08-07), built withGPU_ARCHS=gfx942 - vLLM: commit
4719a9b8f,VLLM_ROCM_USE_AITER=1 - Model:
deepseek-ai/DeepSeek-V4-Flash-0731
Command
vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
--tensor-parallel-size 8 --gpu-memory-utilization 0.9 --max-model-len 8192 \
--kv-cache-dtype fp8 --tokenizer-mode deepseek_v4 --enforce-eager
Error
All 8 TP workers crash identically during profile_run's dummy forward pass:
File ".../aiter/ops/mhc.py", line 350, in mhc_pre
comb_mix = torch.empty(m, hc_mult, hc_mult, dtype=dtypes.fp32, device=device)
torch.AcceleratorError: CUDA error: invalid configuration argument
hc_mult=4 (from the model's config.json). Call path: vllm/models/deepseek_v4/amd/model.py -> hc_pre -> MHCPreOp.forward_hip -> torch.ops.vllm.mhc_pre_aiter -> aiter/ops/mhc.py:mhc_pre.
Reproduced identically (same line, same error) under both HIP_LAUNCH_BLOCKING=1 CUDA_LAUNCH_BLOCKING=1 and AMD_SERIALIZE_KERNEL=3, ruling out simple async-error misattribution.
Workaround
Forcing HAS_AITER_MHC = False in vllm/model_executor/layers/mhc.py routes MHCPreOp/MHCPostOp.forward_hip to forward_native instead, and the model loads and proceeds past this point.
Related
- vllm-project/vllm#42876 -- DeepSeek-V4 on MI350X: different MHC-path error (TileLang PDL), same op family
- vllm-project/vllm#46796 -- DeepSeek-V4 on NVIDIA B300: same error text ("invalid configuration argument") in the same MHC/hc_prenorm kernel family, different backend (DeepGEMM, not AITER)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the DeepSeek-V4-Flash-0731 profile_run with the documented vLLM command, then start at aiter/ops/mhc.py:350 and trace the call path through vllm/models/deepseek_v4/amd/model.py and vllm/model_executor/layers/mhc.py. Compare the AITER path with the native fallback; done means the MI300X run proceeds past MHCPreOp without the invalid configuration error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100