mudler / mudler/vllm.cpp

test_backend_cross_device: MoeSiluMul bf16 exactness reds on gfx1200 ROCm too, the ROCm counterpart of #1802

Open Beginner friendly
#1,954 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

test_backend_cross_device's MoeSiluMul matches the CPU oracle within NMSE <= 5e-4 case fails on ROCm gfx1200, at :2063,
CHECK(got == ref_b). This is the ROCm counterpart of the bookkeeping
#1802 did for sm_110,
which already carries the identical test name and the identical failing
assertion for CUDA.

#907 recorded GB10's own
red sweep and is related but distinct, not a second match for the same
test. It documents test_cuda_ops's "CUDA silu_and_mul matches CPU" case
failing at 439 of 440 assertions, a different test and a different
assertion in the same defect family (silu-and-mul last-digit numerics), not
the same test or the same assertion test_backend_cross_device fails here.

No entry for gfx1200 or for ROCm exists anywhere in .agents/environment.md.

Confirmed pre-existing, not introduced by a change in flight

Found during the fresh review of
row/ROCM-KQUANT-NWARPS-DECODE.
The review reverted only that row's two changed files back to the parent
commit, in a scratch copy, and rebuilt. The failure reproduces byte for byte:

parent commit reviewed commit
cases 24 passed, 1 failed of 25 25 passed, 1 failed of 26
assertions 1 failed of 80195 1 failed of 80253

The one new case and 58 new assertions the row adds all pass. The delta
between the two rows is exactly the row's own test, nothing else.

Whether this is the same root cause as #1802's CUDA entry

Not established here. #1802 records the CUDA sm_110 vectors as differing "in
the last digit of a handful of elements," consistent with an ordinary bf16
rounding difference in a hardware-specific kernel implementation. ROCm's
kernel is a separate implementation from CUDA's, so the two reds could be
independent tolerance gaps in the same test rather than one shared defect.
Nobody has compared the actual mismatching elements across the two backends.

What is NOT established

  • No comparison of which elements mismatch on ROCm against which elements
    mismatch on CUDA.
  • No attribution to a specific MoeSiluMul kernel line on either backend.
  • No speed or correctness claim beyond "this one assertion currently fails."

Owed

Add a gfx1200 row to .agents/environment.md's known-red table, mirroring
the existing sm_110 and GB10 rows, so a future gfx1200 run does not
re-report this as new.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Update .agents/environment.md, using the existing sm_110 and GB10 known-red rows as the model. Confirm the gfx1200 ROCm failure details from issue #1954, then add a gfx1200 row describing the failing test and assertion. Done means future gfx1200 runs no longer report this known red as new.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.