mudler / mudler/vllm.cpp

HIP builds fail to link when ccache is present in the container, and link without it

Open
#2,506 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ROCM

A HIP build of libvllm.so links when ccache is absent from the container and
fails to link when it is present. Three runs on strix:gpu0 (gfx1151, ROCm
7.2.4), same source archive (sha256 a9caa146...), same host, same worker:

rc job ccache installed + CCACHE_DIR set vllm-cli link
0d94bc20-4c86-4138-8592-dbaf6baf7a11 no rc=0, binary built and ran
529b9edc-d4f0-4856-866c-45613d6e35a2 yes rc=1
7894a2bc-e469-4cc7-933f-b99ba415cd61 yes rc=1, identical

The failure

Linking vllm-cli against libvllm.so.0.0.3, with the HIP runtime and hipBLASLt
entry points undefined together:

/usr/bin/ld: libvllm.so.0.0.3: undefined reference to `hipSetDevice@hip_4.2'
/usr/bin/ld: libvllm.so.0.0.3: undefined reference to `hipEventCreateWithFlags@hip_4.2'
/usr/bin/ld: libvllm.so.0.0.3: undefined reference to `__hipUnregisterFatBinary@hip_4.2'
/usr/bin/ld: libvllm.so.0.0.3: undefined reference to `hipGraphGetNodes@hip_4.4'
/usr/bin/ld: libvllm.so.0.0.3: undefined reference to `hipblasLtMatrixLayoutCreate'
/usr/bin/ld: libvllm.so.0.0.3: undefined reference to `hipblasLtMatmulAlgoGetHeuristic'
collect2: error: ld returned 1 exit status

What has been ruled out

  • Stale build directory. Every run did rm -rf on both the source and build
    directories first. The second ccache run was added specifically to test this
    and reproduced the failure.
  • Different CMake resolution. The configure logs are identical on every ROCm
    line. All three resolve ROCm hipBLAS: /opt/rocm/lib/libhipblas.so and
    ROCm hipBLASLt: /opt/rocm/lib/libhipblaslt.so, and all three print
    ROCm backend: ENABLED for arch(es) [gfx1151].
  • A missing package. hipblaslt-dev was installed before configure in all
    three (see #2499), and the header and libhipblaslt.so.1.2.70204 were verified
    present.
  • Project conditionals. target_link_libraries(vllm PUBLIC ...) for
    amdhip64, hipBLAS and hipBLASLt are unconditional inside if(VLLM_CPP_HIP)
    (CMakeLists.txt:1747-1775), so nothing the project itself gates explains a
    link line losing them.
  • VLLM_CPP_BUILD_TESTS. Tested explicitly as the candidate discriminator
    and refuted: the option already defaults to ON (CMakeLists.txt:58), so
    passing it changes nothing, and the run that passed it failed identically.

What is not established

The mechanism. The correlation is clean across three runs, but this is an
observation, not a diagnosis, and no claim is made about why ccache would
affect the link line of a target whose link libraries are unconditional. Plausible
directions someone should check rather than assume: whether apt's
/usr/lib/ccache shims reach the compiler search path used for HIP device
compilation, and whether cached HIP objects lose the fat-binary registration that
pulls the runtime into DT_NEEDED.

Why it is worth an issue

/workspace/ccache is shared fleet infrastructure. Any ROCm row that reaches for
it to shorten a build on a leased box will hit this, and the failure surfaces
~570 TUs in as a wall of undefined references rather than as anything naming
ccache.

Acceptance

  • The mechanism is identified, or the correlation is shown to be something else.
  • A HIP build either works with ccache or refuses it at configure time naming the
    reason.

Found while bringing up GlmMoeDsaForCausalLM on this board (#2498). Worked
around by not installing ccache.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with CMakeLists.txt lines 58 and 1747-1775, then compare the HIP compiler and linker commands from runs with and without ccache. Check whether /usr/lib/ccache shims or cached HIP objects affect runtime linkage, using the recorded ROCm configuration and failure output. Done means the mechanism is identified, or HIP builds reject ccache at configure time with a reason.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
backend, build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.