Instinct serving on RHEL 9.6 requires manual vLLM, OpenMPI, and amdsmi workarounds

Thank you for reviewing the report.

Summary

On RHEL 9.6 with AMD Instinct MI300A (gfx943 / gfx94X-dcgpu), the documented happy path:

curl -fsSL https://raw.githubusercontent.com/ROCm/rocm-cli/main/install.sh | sh -s -- nightly
rocm install sdk
rocm serve qwen

does not reach a working OpenAI-compatible endpoint without several manual workarounds. SDK install succeeds and setup is marked completed, but vLLM engine auto-install fails silently enough that users only discover the problem at rocm serve time.

After manual fixes (detailed below), serving works: http://127.0.0.1:11435/v1 with Qwen/Qwen3-4B-Instruct-2507, chat smoke test OK.

Environment

• OS: RHEL 9.6 (Plow), kernel 5.14.0-570.12.1.el9_6
• Hardware: AMD Instinct MI300A, amdgpu driver loaded (/dev/kfd present)
• rocm-cli: 0.1.0 (nightly channel)
• Managed runtime: TheRock 7.13.0 / gfx94X-dcgpu (release-wheel-gfx94x-dcgpu-7-13-0)
• Starting state: no /opt/rocm, fresh user install

---

Independent clean reproduction

The failure was reproduced a second time in a fresh RHEL 9.6 UBI container with
/dev/kfd and /dev/dri passed through:

registry.access.redhat.com/ubi9/ubi@sha256:dec374e05cc13ebbc0975c9f521f3db6942d27f8ccdf06b180160490eef8bdbc

The clean run detected RHEL 9.6, MI300A, gfx943, and gfx94X-dcgpu
correctly. rocm install sdk installed TheRock 7.13 / torch 2.11, then
reproduced both the OpenMPI package failure and vLLM/torch resolution failure.
The command nevertheless exited 0, and rocm setup status reported
completed while rocm engines list reported vLLM not found.
rocm serve qwen then exited 1, and its failed service remained
starting.

---

Why manual workarounds are needed

There are four independent gaps between what rocm install sdk delivers and what vLLM serving actually requires on RHEL + TheRock 7.13:

1. vLLM wheel index / PyTorch version skew

rocm install sdk installs torch 2.11.0+rocm7.13.0 from TheRock 7.13.0 wheels, then auto-installs vLLM from:

https://wheels.vllm.ai/rocm/0.23.0/rocm723

That index pins torch==2.10.0+git8514f05, which conflicts with the managed runtime. uv fails with:

× No solution found when resolving dependencies:
╰─▶ Because torch==2.10.0+git8514f05 has no wheels with a matching platform
    tag (e.g., `manylinux_2_34_x86_64`) ...
hint: Wheels are available for torch (v2.10.0+git8514f05) on manylinux_2_35_x86_64

Why a workaround is needed: rocm-cli does not select a vLLM wheel index aligned with TheRock 7.13 / torch 2.11. Setup still reports success.

Workaround used on the successfully recovered host: install the vLLM wheel
from a newer index that matches RHEL glibc 2.34 while preserving the TheRock
ROCm torch packages:

UV=~/.rocm/tools/uv/latest/uv-x86_64-unknown-linux-gnu/uv
PY=~/.rocm/runtimes/wheel/release-wheel-gfx94x-dcgpu-7-13-0/bin/python
VLLM_WHEEL='https://wheels.vllm.ai/rocm/6e448d0ea9bf3d88d898b65449ca6dc2aec170ac/vllm-0.27.1%2Brocm723-cp312-cp312-manylinux_2_34_x86_64.whl'
$UV pip install --python $PY --no-deps "$VLLM_WHEEL"

Verify torch remains ROCm: 2.11.0+rocm7.13.0, hip 7.13.99004.

Important: --no-deps is not a standalone clean-install recipe. In a
second fresh SDK environment, the wheel installed but import vllm failed
immediately with:

ModuleNotFoundError: No module named 'packaging'

The recovered host already had vLLM's non-torch dependencies from an earlier
resolver attempt. Recovery required restoring the TheRock torch packages,
removing CUDA torchcodec, and then replacing only the vLLM package with the
ROCm wheel. rocm-cli needs a supported resolver policy that installs vLLM's
non-torch dependencies while pinning the managed ROCm torch/vision/audio/triton
stack.

Anti-pattern: uv pip install vllm --index-strategy unsafe-best-match without pinning torch replaces ROCm torch with CUDA wheels (torch 2.13.0+cu130, hip None).

---

2. OpenMPI not installable via hardcoded dnf install openmpi

During vLLM auto-install, rocm-cli runs:

sudo dnf install -y openmpi

On RHEL 9.6 this fails:

No match for argument: openmpi
Error: Unable to find a match: openmpi
warning: OpenMPI install failed ...
warning: continuing vLLM install ...

Even when rocm-cli's own message mentions RHEL paths (/usr/lib64/openmpi, module load mpi/openmpi-x86_64), the install command uses the wrong package name and does not handle missing base/AppStream repos.

Why a workaround is needed: vLLM's ROCm torch stack expects MPI libraries at runtime; rocm-cli's preflight checks for libmpi.so, libmpi_cxx.so.40, and mpirun under /usr/lib64/openmpi/.

Workaround that worked: build OpenMPI 4.1.6 from source and install into the RHEL-expected layout:

# build to ~/.local/openmpi (configure --disable-mpi-fortran)
sudo mkdir -p /usr/lib64/openmpi/lib /usr/lib64/openmpi/bin
sudo cp -a ~/.local/openmpi/lib/libmpi.so* \
           ~/.local/openmpi/lib/libopen-pal.so* \
           ~/.local/openmpi/lib/libopen-rte.so* \
           /usr/lib64/openmpi/lib/
sudo ln -sf libmpi.so.40 /usr/lib64/openmpi/lib/libmpi_cxx.so.40
sudo cp -a ~/.local/openmpi/bin/mpirun /usr/lib64/openmpi/bin/

Upstream OpenMPI 4.x does not ship libmpi_cxx; rocm-cli's check requires the RHEL-style symlink.

---

3. Missing amdsmi Python package for vLLM ROCm platform detection

After OpenMPI is resolved, vLLM fails at startup with:

RuntimeError: Failed to infer device type

With VLLM_LOGGING_LEVEL=DEBUG:

ROCm platform is not available because: No module named 'amdsmi'

The managed TheRock runtime does include libamd_smi.so under _rocm_sdk_devel/lib, but rocm-cli does not install the matching Python amdsmi package during engine setup.

Workaround that worked:

$UV pip install --python $PY \
  'https://wheels.vllm.ai/rocm/6e448d0ea9bf3d88d898b65449ca6dc2aec170ac/amdsmi-26.2.2%2Bc2d9476115-py3-none-any.whl'

After this, vLLM logs Confirmed ROCm platform is available / Automatically detected platform rocm.

---

4. Secondary: broken torchcodec from failed install attempts

If users retry vLLM install with unsafe-best-match, CUDA torchcodec can land in the managed venv and break from vllm import LLM with:

OSError: Could not load this library: .../torchcodec/libtorchcodec_image.so

(ldd shows missing libtorch.so, libcudart.so.13, etc.)

Workaround: uv pip uninstall torchcodec

---

Observed UX gaps (related)

Issue	Impact
rocm setup status → completed after vLLM auto-install failure	Users think they are ready to serve
rocm diagnose surfaces false positives (amdgpu not loaded, user not in render group) on a working GPU host	Real blockers (vLLM, OpenMPI, amdsmi) not diagnosed
Failed rocm serve leaves service in starting until manually stopped	Confusing recovery
rocm examine / dash show Unknown GPU without system amd-smi	Telemetry degraded even when GPU works

---

Suggested fixes

1. Align vLLM wheel index with installed TheRock/torch version (7.13 → torch 2.11), prefer manylinux_2_34 wheels on RHEL 9 / glibc 2.34.
2. RHEL-aware OpenMPI install: correct package/module names, repo prerequisites, or documented source-build fallback; don't hardcode openmpi.
3. Auto-install amdsmi (matching wheel) and ensure libamd_smi.so from the managed runtime is on the vLLM launch LD_LIBRARY_PATH.
4. Block or warn on setup completion when vLLM engine install fails; surface actionable next steps in rocm examine / rocm diagnose.
5. Pin torch during any vLLM dependency resolution so ROCm torch is never replaced by CUDA wheels.

---

Verification after workarounds

rocm serve qwen
# ~90s model load
curl -s http://127.0.0.1:11435/v1/models
curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-4B-Instruct-2507","messages":[{"role":"user","content":"Say hello in one word."}],"max_tokens":10}'
# → "Hi"

---

Repro notes

• Fresh install path only; no pre-existing /opt/rocm.
• RHEL base repos were not registered on the test host (only EPEL + Docker); OpenMPI dnf failure may partially reflect repo state, but the openmpi package name and continue-after-failure behavior are still bugs on registered RHEL.
• Hardware: MI300A (Instinct); engine selection to vLLM was correct throughout.

Start with the documented RHEL 9.6 reproduction using rocm install sdk, rocm setup status, rocm engines list, and rocm serve qwen. Trace the vLLM auto-install, OpenMPI handling, and amdsmi detection paths described in the report. Done means setup surfaces failures clearly and the managed runtime can reach the verified OpenAI-compatible endpoint without the listed manual workarounds.