kvcache-ai / kvcache-ai/Mooncake

[Installation]: Wheels compiled with MACA flags cannot be installed on identically configured machines

Open
#2,574 2 comments 0 reactions 1 assignee Claimed by @alogfans View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Installation errors

## Background

This issue was discovered during development and deployment on Muxi GPU platforms. When building the Mooncake Transfer Engine wheel package with MACA compilation parameters enabled, the generated installation package cannot be deployed normally on other devices with the same hardware and software environment configuration.

## Issue Description

When we build the wheel with `-DMACA` compilation flags, the output wheel package installs successfully on other machines with identical Muxi GPU hardware and system environments. However, an error is raised during TE initialization when running Python scripts.

The detailed error log is provided below:

```
terminate called after throwing an instance of 'spdlog::spdlog_ex'
what(): logger with name 'KFI' already exists what(): logger with name 'KFI' already exists Aborted (core dumped)
```

## Root Cause

The `auditwheel repair` command automatically bundles all dependent shared libraries (including MACA runtime libraries `libmccl.so*`, `libmcruntime.so*`) into the wheel package by default during the packaging process.

At runtime, the bundled MACA runtime libraries attempt to initialize their own logger named `KFI` upon loading. However, the Mooncake Transfer Engine framework has already initialized a logger with the exact same `KFI` name in the current process. Since spdlog does not allow creating duplicate loggers with identical names, it throws an `spdlog::spdlog_ex` exception with the message *"logger with name 'KFI' already exists"*, which triggers a process abort during TE initialization

## Proposed Solution
Add MACA-related runtime libraries to the exclusion list of the `auditwheel repair` command in the wheel build script, following the same pattern as the existing excluded third-party libraries.

The modified packaging command is as follows:
```bash
${AUDITWHEEL_CMD} repair ${OUTPUT_DIR}/*.whl \
--exclude libcurl.so* \
--exclude libfabric.so* \
--exclude libefa.so* \
...
--exclude libaccl_barex.so* \
--exclude liburma.so* \
+ --exclude libmccl.so* \
+ --exclude libmcruntime.so*
```

## Environment Information

- Hardware: Muxi GPU
- Build flag: `-DMACA` enabled
- Component: Mooncake Transfer Engine (TE) wheel packaging
- Tool: auditwheel

### Before submitting a new issue...

- [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.