mudler / mudler/vllm.cpp

No CI lane compiles a .hip: every src/vt/rocm change reaches main with zero build coverage, and #2957 landed that way

Open
#3,021 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: -

No lane in .github/workflows/ci.yml compiles a .hip file. The lanes are:

build-test-cpu            build-test-cpu-arm64      build-test-cpu-arm64-full
build-test-vulkan         cuda-arch-features        cuda-fat-build
windows-msvc-cpu          windows-msvc-vulkan       macos-metal-mlx
sanitize-cpu (address,undefined)                    sanitize-cpu (thread)
device-leakage            pr-size                   vulkan-spirv-freshness
drift-check               last-gated-commit         plan

So every change under src/vt/rocm/ — 30-odd translation units — reaches main with no build coverage of any kind. A green rollup on such a PR is not weak evidence; it is zero evidence, because nothing in it read the file.

This is not hypothetical. Both instances are from 2026-09-06.

Instance 1 — #3001 was held, and the manual gate was necessary

src/vt/rocm/rocm_moe_gate_up_swiglu.hip was written by an implementer whose host had no ROCm toolchain, so the TU had been compiled by nobody. Its CI rollup was green on the lanes that ran. I held the merge and queued a compile job on strix:gpu0 (4438b50e), which built it and ran the focused case:

NINJA rc=0
[557/585] Building HIP object .../rocm_moe_gate_up_swiglu.hip.o
SELECTED_CASES=1 ASSERTIONS=50

It happened to build. The point is that nothing in the project would have told us if it had not.

Instance 2 — #2957 landed unverified, and was compiled for the first time AFTERWARDS

ac77de7b4 merged the silu/gelu gate-dtype narrowing, touching src/vt/rocm/rocm_dense_basic.hip and src/vt/rocm/rocm_moe_router.hip. Its rollup carried one entry (plan).

A job queued for an unrelated attribution question (74a5a81c) built the merge commit on gfx1151 hours later and ran its focused case: 3 test cases, 7061/7061 assertions, 0 skipped, with a discrimination arm showing 6 bf16 := true CHECKs going red without the fix. Good result — and it arrived after the code was already on main. Had it not compiled, main would have been broken for every ROCm consumer with no gate anywhere to catch it.

This is #2950 with a specific, dated instance rather than a general worry.

What would close it, in increasing order of cost

  1. A refusal, which needs no hardware. A checker that fails when a change touches src/vt/rocm/** and no ROCm build ran, pointing at the manual recipe. It converts silent absence into a visible PENDING, which is what AGENTS.md §Gates asks for when a gate cannot run. This alone would have flagged both instances above.
  2. A compile-only lane, no GPU: install the ROCm toolchain in the runner and cmake -DVLLM_CPP_HIP=ON -DVLLM_CPP_HIP_ARCHITECTURES=gfx1151 + build. Catches every syntax and semantic error, which is what both instances were actually at risk of. It does not run a kernel and does not need a device.
  3. A device lane on strix:gpu0, running the focused cross-device cases. Real coverage, but it needs a lease and is contended.

(2) is the one worth doing. The recipe is already proven — /mnt/nas_share/rc/rocmbuild-3001/job.sh configures and builds test_backend_cross_device for gfx1151 in minutes, and docs/bench-evidence/ carries several working ROCm build recipes.

One trap to carry into whatever is built

A doctest filter that matches no case exits 0 and prints Status: SUCCESS! over 0 test cases | 0 assertions. The first attempt at #3001's device gate did exactly that and would have merged it on a gate that measured nothing. Any lane added here must read both counts and treat a zero as VOID, and ideally derive the expected assertion count from the test source so a self-skip is distinguishable from a real run rather than merely non-zero.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .github/workflows/ci.yml and AGENTS.md §Gates, then inspect /mnt/nas_share/rc/rocmbuild-3001/job.sh and the ROCm recipes in docs/bench-evidence/. Add coverage for src/vt/rocm/ using the stated HIP CMake configuration, and ensure a zero-case or zero-assertion result is treated as void rather than success.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, github-actions
Domain
build-system, ci-cd
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.