ROCm: DFlash2 reaches host fallbacks and hangs gfx1151
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-ROCM
Qwen3.8-27B with the published DFlash2 safetensors draft does not complete on strix:gpu0 (gfx1151, ROCm 7.2.4).
The plain target completes 32 greedy tokens. The DFlash2 arm reaches three CPU reference-tier ops and then the GPU reports a hang:
[vt reference-tier] op=IndexSelect device=rocm has NO native kernel
[vt reference-tier] op=IndexCopy device=rocm has NO native kernel
[vt reference-tier] op=DFlashGroupedConv device=rocm has NO native kernel
HW Exception by GPU node-1 ... reason :GPU Hang
The DFlash2 process exits 134 before its first completion. test_dflash2_runner_reach is also red on the same HIP build. Its D9 case reports zero differing blocks after changing the selector scalars. One repeated run also failed the W11 call-count and output-size checks.
GetOp calls Backend::FlushPending() before a CPU reference-tier kernel reads managed device memory. RocmBackend does not override that method, so it inherits the no-op. Metal and Vulkan both override it. This is the first root-cause candidate for the hang and the unstable runner output.
A safe run is not enough for a performance result. The production DFlash2 chain also needs native ROCm implementations for every op that the full provider census reports. The known static set includes row gather/scatter, grouped convolution, top-k pairs, selector edges, path walk, and the block-attention variants.
Acceptance:
- A device test proves that
RocmBackend::FlushPending()drains queued HIP work before a host fallback. - The real Qwen3.8-27B plus DFlash2 arm completes greedy generation without a GPU hang.
- The focused DFlash2 suites pass on
gfx1151. VT_OP_PROVIDER_STATS=1reports zero CPU reference-tier hits for the measured DFlash2 workload.- A repeated local-checkpoint benchmark records clock state and warm-leg throughput.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at RocmBackend::FlushPending and GetOp, then inspect the device test coverage around host fallbacks. Run test_dflash2_runner_reach and the focused DFlash2 suites on gfx1151. Done means queued HIP work is drained, Qwen3.8-27B DFlash2 completes without a hang, tests pass, and provider stats show no CPU reference-tier hits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, machine-learning, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100