No device value names ROCm: the C ABI offers only auto/cpu/cuda, so an AMD box cannot pin its accelerator
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-ROCM
There is no way to ask for the ROCm device by name. vllm-cli --device rocm is a
usage error on a machine whose only accelerator is a Radeon:
vllm-cli: unknown --device 'rocm' (expected auto, cpu, or cuda)
Measured on strix:gpu0 (gfx1151, ROCm 7.2.4) against a HIP build that had
just reported ROCm backend: ENABLED for arch(es) [gfx1151] at configure time,
rc job 0d94bc20-4c86-4138-8592-dbaf6baf7a11.
Where it comes from
The CLI is a thin client and is not really the problem.
examples/cli/main.cpp:143-150 maps three names, because that is what the C ABI
offers. include/vllm.h:113-120 documents vllm_model_params.device as:
0=auto the accelerator-first platform probe
1=cpu force the CPU queue even on an accelerator build
2=cuda REQUIRE the CUDA platform; when it is absent the load FAILS
2 is explicitly not a generic "accelerator" value -- it names CUDA and refuses
to be silently substituted, mirroring vllm/config/device.py:61-66. So on an AMD
box auto is the only route to the GPU.
Why that is worse than an inconvenience
The load-time device walk already knows about the other backends: it resolves
over {kCUDA, kROCM, kXPU, kVULKAN, kMETAL, kTENSTORRENT, kCPU}
(src/vllm/entrypoints/model_loader.cpp:374), and vt::DeviceType has carried
kROCM since the backend landed. The engine can reach ROCm. The public surface
cannot say so.
Two consequences:
- A user cannot pin the accelerator. If the platform fails to register --
a driver problem, a container without/dev/kfd, a build whose HIP TUs were
dropped --autoquietly resolves to the CPU queue and the model still runs,
slowly. The difference between "ran on the Radeon" and "fell back to the CPU"
is only visible by reading the log.1=cpuexists precisely so a user can
force the slow path deliberately; there is no counterpart for forcing the
fast one, so the failure is silent in the direction that matters. - It contradicts the shared-seam rule. AGENTS.md §"Shared seams" requires
every shipped capability to be reachable throughinclude/vllm.h, with
examples and servers as thin clients. ROCm is a shipped backend --
docs/ROCM.mddocuments bring-up on five gfx architectures -- and it has no
ABI selector. The same is true of Metal, Vulkan, XPU and Tenstorrent, each of
which is in that device walk and none of which can be named.
What a fix owes
Appending device values is an ABI version bump with a documented note, exactly as
v14 added the field in the first place, and the enum should mirror what
vt::DeviceType and the loader walk already agree on. It also needs a decision
this issue does not make: whether an explicitly named accelerator that is absent
FAILS the load, as 2=cuda does today, which is the behaviour that makes the
flag worth having.
Worth pairing with the CLI usage string and docs/ROCM.md, whose Quickstart
currently cannot show a command that names this backend.
Acceptance
- A caller can name ROCm through
include/vllm.hand throughvllm-cli. - Naming an accelerator that is absent fails the load rather than substituting.
docs/ROCM.mdshows a command that pins the device.0=auto,1=cpuand2=cudakeep their exact current behaviour.
Found while bringing up GlmMoeDsaForCausalLM on this board (#2498), which is
what the run above was trying to do.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with include/vllm.h:113-120 and trace the existing device mapping in examples/cli/main.cpp:143-150, then read src/vllm/entrypoints/model_loader.cpp:374 and vllm/config/device.py:61-66. Review AGENTS.md's Shared seams guidance and docs/ROCM.md. Done means named-device selection works through the ABI and CLI, absent accelerators fail, existing values remain unchanged, and the ROCm quickstart shows the command.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- api, backend, cli, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100