HIP builds fail to link when ccache is present in the container, and link without it
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-ROCM
A HIP build of libvllm.so links when ccache is absent from the container and
fails to link when it is present. Three runs on strix:gpu0 (gfx1151, ROCm
7.2.4), same source archive (sha256 a9caa146...), same host, same worker:
| rc job | ccache installed + CCACHE_DIR set |
vllm-cli link |
|---|---|---|
0d94bc20-4c86-4138-8592-dbaf6baf7a11 |
no | rc=0, binary built and ran |
529b9edc-d4f0-4856-866c-45613d6e35a2 |
yes | rc=1 |
7894a2bc-e469-4cc7-933f-b99ba415cd61 |
yes | rc=1, identical |
The failure
Linking vllm-cli against libvllm.so.0.0.3, with the HIP runtime and hipBLASLt
entry points undefined together:
/usr/bin/ld: libvllm.so.0.0.3: undefined reference to `hipSetDevice@hip_4.2'
/usr/bin/ld: libvllm.so.0.0.3: undefined reference to `hipEventCreateWithFlags@hip_4.2'
/usr/bin/ld: libvllm.so.0.0.3: undefined reference to `__hipUnregisterFatBinary@hip_4.2'
/usr/bin/ld: libvllm.so.0.0.3: undefined reference to `hipGraphGetNodes@hip_4.4'
/usr/bin/ld: libvllm.so.0.0.3: undefined reference to `hipblasLtMatrixLayoutCreate'
/usr/bin/ld: libvllm.so.0.0.3: undefined reference to `hipblasLtMatmulAlgoGetHeuristic'
collect2: error: ld returned 1 exit status
What has been ruled out
- Stale build directory. Every run did
rm -rfon both the source and build
directories first. The second ccache run was added specifically to test this
and reproduced the failure. - Different CMake resolution. The configure logs are identical on every ROCm
line. All three resolveROCm hipBLAS: /opt/rocm/lib/libhipblas.soand
ROCm hipBLASLt: /opt/rocm/lib/libhipblaslt.so, and all three print
ROCm backend: ENABLED for arch(es) [gfx1151]. - A missing package.
hipblaslt-devwas installed before configure in all
three (see #2499), and the header andlibhipblaslt.so.1.2.70204were verified
present. - Project conditionals.
target_link_libraries(vllm PUBLIC ...)for
amdhip64, hipBLAS and hipBLASLt are unconditional insideif(VLLM_CPP_HIP)
(CMakeLists.txt:1747-1775), so nothing the project itself gates explains a
link line losing them. VLLM_CPP_BUILD_TESTS. Tested explicitly as the candidate discriminator
and refuted: the option already defaults toON(CMakeLists.txt:58), so
passing it changes nothing, and the run that passed it failed identically.
What is not established
The mechanism. The correlation is clean across three runs, but this is an
observation, not a diagnosis, and no claim is made about why ccache would
affect the link line of a target whose link libraries are unconditional. Plausible
directions someone should check rather than assume: whether apt's
/usr/lib/ccache shims reach the compiler search path used for HIP device
compilation, and whether cached HIP objects lose the fat-binary registration that
pulls the runtime into DT_NEEDED.
Why it is worth an issue
/workspace/ccache is shared fleet infrastructure. Any ROCm row that reaches for
it to shorten a build on a leased box will hit this, and the failure surfaces
~570 TUs in as a wall of undefined references rather than as anything naming
ccache.
Acceptance
- The mechanism is identified, or the correlation is shown to be something else.
- A HIP build either works with ccache or refuses it at configure time naming the
reason.
Found while bringing up GlmMoeDsaForCausalLM on this board (#2498). Worked
around by not installing ccache.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with CMakeLists.txt lines 58 and 1747-1775, then compare the HIP compiler and linker commands from runs with and without ccache. Check whether /usr/lib/ccache shims or cached HIP objects affect runtime linkage, using the recorded ROCm configuration and failure output. Done means the mechanism is identified, or HIP builds reject ccache at configure time with a reason.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp
- Domain
- backend, build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100