mudler / mudler/vllm.cpp

No device value names ROCm: the C ABI offers only auto/cpu/cuda, so an AMD box cannot pin its accelerator

Open
#2,505 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ROCM

There is no way to ask for the ROCm device by name. vllm-cli --device rocm is a
usage error on a machine whose only accelerator is a Radeon:

vllm-cli: unknown --device 'rocm' (expected auto, cpu, or cuda)

Measured on strix:gpu0 (gfx1151, ROCm 7.2.4) against a HIP build that had
just reported ROCm backend: ENABLED for arch(es) [gfx1151] at configure time,
rc job 0d94bc20-4c86-4138-8592-dbaf6baf7a11.

Where it comes from

The CLI is a thin client and is not really the problem.
examples/cli/main.cpp:143-150 maps three names, because that is what the C ABI
offers. include/vllm.h:113-120 documents vllm_model_params.device as:

0=auto  the accelerator-first platform probe
1=cpu   force the CPU queue even on an accelerator build
2=cuda  REQUIRE the CUDA platform; when it is absent the load FAILS

2 is explicitly not a generic "accelerator" value -- it names CUDA and refuses
to be silently substituted, mirroring vllm/config/device.py:61-66. So on an AMD
box auto is the only route to the GPU.

Why that is worse than an inconvenience

The load-time device walk already knows about the other backends: it resolves
over {kCUDA, kROCM, kXPU, kVULKAN, kMETAL, kTENSTORRENT, kCPU}
(src/vllm/entrypoints/model_loader.cpp:374), and vt::DeviceType has carried
kROCM since the backend landed. The engine can reach ROCm. The public surface
cannot say so.

Two consequences:

  1. A user cannot pin the accelerator. If the platform fails to register --
    a driver problem, a container without /dev/kfd, a build whose HIP TUs were
    dropped -- auto quietly resolves to the CPU queue and the model still runs,
    slowly. The difference between "ran on the Radeon" and "fell back to the CPU"
    is only visible by reading the log. 1=cpu exists precisely so a user can
    force the slow path deliberately; there is no counterpart for forcing the
    fast one, so the failure is silent in the direction that matters.
  2. It contradicts the shared-seam rule. AGENTS.md §"Shared seams" requires
    every shipped capability to be reachable through include/vllm.h, with
    examples and servers as thin clients. ROCm is a shipped backend --
    docs/ROCM.md documents bring-up on five gfx architectures -- and it has no
    ABI selector. The same is true of Metal, Vulkan, XPU and Tenstorrent, each of
    which is in that device walk and none of which can be named.

What a fix owes

Appending device values is an ABI version bump with a documented note, exactly as
v14 added the field in the first place, and the enum should mirror what
vt::DeviceType and the loader walk already agree on. It also needs a decision
this issue does not make: whether an explicitly named accelerator that is absent
FAILS the load, as 2=cuda does today, which is the behaviour that makes the
flag worth having.

Worth pairing with the CLI usage string and docs/ROCM.md, whose Quickstart
currently cannot show a command that names this backend.

Acceptance

  • A caller can name ROCm through include/vllm.h and through vllm-cli.
  • Naming an accelerator that is absent fails the load rather than substituting.
  • docs/ROCM.md shows a command that pins the device.
  • 0=auto, 1=cpu and 2=cuda keep their exact current behaviour.

Found while bringing up GlmMoeDsaForCausalLM on this board (#2498), which is
what the run above was trying to do.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with include/vllm.h:113-120 and trace the existing device mapping in examples/cli/main.cpp:143-150, then read src/vllm/entrypoints/model_loader.cpp:374 and vllm/config/device.py:61-66. Review AGENTS.md's Shared seams guidance and docs/ROCM.md. Done means named-device selection works through the ABI and CLI, absent accelerators fail, existing values remain unchanged, and the ROCm quickstart shows the command.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend, cli, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.