test_backend_cross_device: MoeRouterTopK flakes intermittently on gfx1200
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-ROCM
tests/vt/test_backend_cross_device.cpp:2304, TEST_CASE("MoeRouterTopK matches the CPU oracle (f32 and bf16 logits)"), failed once during work on a fresh implementer's row (row/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-w1, unrelated to MoE routing), on isravale (RX 9060 XT, gfx1200, ROCm 7.2.3):
tests/vt/test_backend_cross_device.cpp:2351: ERROR: CHECK( Nmse(ref_w, dw.Download()) <= kNmseTol ) is NOT correct!
values: CHECK( 0.214825 <= 0.0005 )
plus mismatched expert indices in the same case.
Reproduction rate is low and I want to record it honestly rather than overstate it. The implementer who found it saw 1 failure in 12 full-test_backend_cross_device-binary runs with their (unrelated) change applied, and 1 in 12 with that change stashed at the pristine base commit c6e1092dffc5f3484d9c8d24ad3b4e3f7b7531cc — ruling their own diff out as the cause. I (operator) then ran the isolated case 10/10 clean and the full binary 6/6 clean, so 0 reproductions in 16 of my own runs. Net: 2 failures observed in 28 total runs across two sessions (~7%), never in isolation from either session — it may require the full binary's accumulated device/allocator state to trigger, which is why isolated single-case runs may under-sample it.
Confirmed unrelated to any in-flight PR: the test case and vt::MoeRouterTopK's kernel source are both absent from row/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-w1's diff, and the test case is confirmed present, unmodified, at that branch's pre-reconciliation head 22800690b (predates all work on that row).
Distinct from #3119 (causal conv1d, a different test case, much higher ~37% isolated reproduction rate) and from #1954 (MoeSiluMul). Given this project's position that no device-side ASan exists for RDNA/gfx12, and that this session alone has found two independent classes of adjacent-but-different intermittent ROCm numerical flakes, this may be worth checking whether all three share a root cause (a common allocator/queue-state issue) rather than being investigated as three unrelated kernels — but that is a hypothesis, not a finding, and needs someone to actually look.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/vt/test_backend_cross_device.cpp:2304, the MoeRouterTopK test, and compare isolated runs with repeated full test_backend_cross_device binary runs on gfx1200. Investigate the intermittent NMSE and expert-index mismatches, including whether allocator or queue state relates this case to the other reported ROCm flakes. Done means the cause is identified and a fix or actionable conclusion is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100