mudler / mudler/vllm.cpp

VLLM_CPP_HIP=ON configures without hipBLASLt and then fails the build 539 TUs later

Open
#2,499 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ROCM

VLLM_CPP_HIP=ON configures successfully on a ROCm install that has hipBLAS but
no hipBLASLt, then fails the build 539 TUs later.

What happens

CMakeLists.txt:1760-1775 treats the two math libraries asymmetrically, and
deliberately so: hipBLAS is required with a FATAL_ERROR, hipBLASLt is
best-effort and its if(...) block simply does not run when the header and
library are absent.

But src/vt/rocm/rocm_matmul_hipblaslt.hip is in the unconditional ROCm source
list (CMakeLists.txt:1704, again at 1727) and includes the header with no
guard:

#include <hipblaslt/hipblaslt.h>   // src/vt/rocm/rocm_matmul_hipblaslt.hip:15

So the optional detection buys nothing. Configure reports only

-- ROCm hipBLAS: /opt/rocm/lib/libhipblas.so

with no hipBLASLt line at all, prints Configuring done / Generating done, and
the build then dies with

/src/vt/rocm/rocm_matmul_hipblaslt.hip:15:10: fatal error:
    'hipblaslt/hipblaslt.h' file not found
1 error generated when compiling for gfx1151.
ninja: build stopped: subcommand failed.

after 538 of 570 objects have already been compiled.

Measured on

strix:gpu0, gfx1151, ROCm 7.2.4, HIP 7.2.53211-97f5574fe2, AMD clang
22.0.0git, inside the leased rc container. rc job 8293713b-1c93-4a5b-aa0c-a06b6245d892,
log /mnt/nas_share/rc/glm53-rocm/out3/build_cli.log:601-604, configure log
/mnt/nas_share/rc/glm53-rocm/out3/cmake.log:26.

The leased worker image carries the ROCm runtime and hipBLAS but not the
hipBLASLt development package, so this is the default state of a fresh lease on
this box rather than an unusual install.

Why it is worth fixing rather than documenting

The runtime path is already opt-in and defaults OFF —
src/vt/rocm/rocm_matmul_hipblaslt.hip:253-256 gates the heuristic on
VT_ROCM_HIPBLASLT=1 and the comment records that it "aborted on gfx1201 in
lab". So a build that cannot reach the library at all still pays for it with a
hard failure, and pays late.

Two shapes would both work, and choosing between them is a ROCm-build decision
that should be made deliberately rather than by whoever hits it next:

  1. Make hipBLASLt required, with the same FATAL_ERROR hipBLAS gets, so the
    refusal names the missing package at configure time instead of at TU 539.
  2. Keep it optional and make the TU match: drop it from the source list when
    VLLM_CPP_HIPBLASLT_INCLUDE is unset, and compile-guard the provider
    registration.

Either way the configure-time report should state which of the two ROCm math
libraries was found, so the build's compiled feature set is readable before the
build rather than inferred from a failure.

Acceptance

  • On a ROCm install without hipBLASLt, -DVLLM_CPP_HIP=ON either refuses at
    configure time naming the package, or configures and builds to completion.
  • The configure log states the resolved hipBLAS and hipBLASLt status either way.
  • No change to a build on an install that has both.

Found while bringing up GlmMoeDsaForCausalLM on this board (#2498).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with CMakeLists.txt:1704, 1727, and 1760-1775, then inspect src/vt/rocm/rocm_matmul_hipblaslt.hip:15 and 253-256. Reproduce configuration and compilation with hipBLAS available but hipBLASLt absent, using the reported logs as reference. Done means the missing-library state is reported during configuration or the build completes, while installs with both libraries remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
66/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.