kvcache-ai / kvcache-ai/Mooncake
[Installation]: Wheels compiled with MACA flags cannot be installed on identically configured machines
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Installation errors
## Background
This issue was discovered during development and deployment on Muxi GPU platforms. When building the Mooncake Transfer Engine wheel package with MACA compilation parameters enabled, the generated installation package cannot be deployed normally on other devices with the same hardware and software environment configuration.
## Issue Description
When we build the wheel with `-DMACA` compilation flags, the output wheel package installs successfully on other machines with identical Muxi GPU hardware and system environments. However, an error is raised during TE initialization when running Python scripts.
The detailed error log is provided below:
```
terminate called after throwing an instance of 'spdlog::spdlog_ex'
what(): logger with name 'KFI' already exists what(): logger with name 'KFI' already exists Aborted (core dumped)
```
## Root Cause
The `auditwheel repair` command automatically bundles all dependent shared libraries (including MACA runtime libraries `libmccl.so*`, `libmcruntime.so*`) into the wheel package by default during the packaging process.
At runtime, the bundled MACA runtime libraries attempt to initialize their own logger named `KFI` upon loading. However, the Mooncake Transfer Engine framework has already initialized a logger with the exact same `KFI` name in the current process. Since spdlog does not allow creating duplicate loggers with identical names, it throws an `spdlog::spdlog_ex` exception with the message *"logger with name 'KFI' already exists"*, which triggers a process abort during TE initialization
## Proposed Solution
Add MACA-related runtime libraries to the exclusion list of the `auditwheel repair` command in the wheel build script, following the same pattern as the existing excluded third-party libraries.
The modified packaging command is as follows:
```bash
${AUDITWHEEL_CMD} repair ${OUTPUT_DIR}/*.whl \
--exclude libcurl.so* \
--exclude libfabric.so* \
--exclude libefa.so* \
...
--exclude libaccl_barex.so* \
--exclude liburma.so* \
+ --exclude libmccl.so* \
+ --exclude libmcruntime.so*
```
## Environment Information
- Hardware: Muxi GPU
- Build flag: `-DMACA` enabled
- Component: Mooncake Transfer Engine (TE) wheel packaging
- Tool: auditwheel
### Before submitting a new issue...
- [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
Contributor guide
Assessment
This issue has not been assessed yet.