ROCm / ROCm/aiter

[Bug] mhc_pre "CUDA error: invalid configuration argument" on MI300X (gfx942) during DeepSeek-V4-Flash-0731 profile_run

Open
#4,633 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
565
Forks
585
Avg merge
3d 4h
Merged PRs (30d)
366

Description

Environment

  • GPU: 8x AMD Instinct MI300X (gfx942)
  • ROCm: 7.1.1, PyTorch 2.14.0.dev20260803+rocm7.1
  • aiter: reproduced on both v0.1.19 and main @ 17f24ec6e (2026-08-07), built with GPU_ARCHS=gfx942
  • vLLM: commit 4719a9b8f, VLLM_ROCM_USE_AITER=1
  • Model: deepseek-ai/DeepSeek-V4-Flash-0731

Command

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --tensor-parallel-size 8 --gpu-memory-utilization 0.9 --max-model-len 8192 \
  --kv-cache-dtype fp8 --tokenizer-mode deepseek_v4 --enforce-eager

Error

All 8 TP workers crash identically during profile_run's dummy forward pass:

File ".../aiter/ops/mhc.py", line 350, in mhc_pre
    comb_mix = torch.empty(m, hc_mult, hc_mult, dtype=dtypes.fp32, device=device)
torch.AcceleratorError: CUDA error: invalid configuration argument

hc_mult=4 (from the model's config.json). Call path: vllm/models/deepseek_v4/amd/model.py -> hc_pre -> MHCPreOp.forward_hip -> torch.ops.vllm.mhc_pre_aiter -> aiter/ops/mhc.py:mhc_pre.

Reproduced identically (same line, same error) under both HIP_LAUNCH_BLOCKING=1 CUDA_LAUNCH_BLOCKING=1 and AMD_SERIALIZE_KERNEL=3, ruling out simple async-error misattribution.

Workaround

Forcing HAS_AITER_MHC = False in vllm/model_executor/layers/mhc.py routes MHCPreOp/MHCPostOp.forward_hip to forward_native instead, and the model loads and proceeds past this point.

Related

  • vllm-project/vllm#42876 -- DeepSeek-V4 on MI350X: different MHC-path error (TileLang PDL), same op family
  • vllm-project/vllm#46796 -- DeepSeek-V4 on NVIDIA B300: same error text ("invalid configuration argument") in the same MHC/hc_prenorm kernel family, different backend (DeepGEMM, not AITER)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the DeepSeek-V4-Flash-0731 profile_run with the documented vLLM command, then start at aiter/ops/mhc.py:350 and trace the call path through vllm/models/deepseek_v4/amd/model.py and vllm/model_executor/layers/mhc.py. Compare the AITER path with the native fallback; done means the MI300X run proceeds past MHCPreOp without the invalid configuration error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.