kvcache-ai / kvcache-ai/Mooncake

[build] macOS/Apple Silicon: TCP-only transfer engine path is blocked by unconditional `infiniband/verbs.h` include (and wheel setup.py hard-blocks darwin)

Open
#3,894 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

## Summary

Mooncake's TCP transport cannot even be *compiled* on macOS today, because `tcp_transport.h` unconditionally includes the RDMA verbs header, and the wheel `setup.py` explicitly exits on darwin. With PD-disaggregated serving now reaching Apple-Silicon clusters (e.g. vllm-metal's new experimental PD path: vllm-project/vllm-metal#709), a TCP-only Mooncake client on macOS would be a natural fit for Mac prefill/decode pools — but the build blocks it before any transport logic is reached.

## Evidence (main @ 2026-09-06)

`mooncake-transfer-engine/include/transport/tcp_transport/tcp_transport.h:17-18`:

```cpp
#ifndef TCP_TRANSPORT_H_
#define TCP_TRANSPORT_H_

#include
```

The **TCP** transport header includes `infiniband/verbs.h` unconditionally — on macOS there is no libibverbs/rdma-core, so any TU that pulls this header fails immediately. Per the docs, `protocol=tcp` is a first-class DRAM↔DRAM path that shouldn't need verbs at all (https://kvcache-ai.github.io/Mooncake/getting_started/supported-protocols/).

`mooncake-wheel/setup.py:10-12`:

```python
unsupported_platforms = ["win32", "darwin"] # Still blocking non-Linux builds
if sys.platform in unsupported_platforms:
sys.exit(
f"Error: mooncake does not support {platform.system()} at this time. "
"Please use a supported Linux distribution."
)
```

(Notably the same file already contains pre-written darwin arm64/x86_64 + macosx tag recognition helpers — the guard is the only thing standing in the way of at least experimental wheels.)

Additionally `mooncake-transfer-engine/include/transport/common.h` includes `` (Linux-only libnuma) unconditionally, and `dependencies.sh` covers only rpm/deb distros, so a TCP-only macOS build would need that include gated too.

## Environment

- macOS 15.6 / 26.0, Apple Silicon (M3 Max / M4 Max)
- Attempted: transfer-engine CMake configure + `pip install mooncake-transfer-engine` (no darwin wheel exists on PyPI — only manylinux x86_64/aarch64)

## Suggestion (happy to PR)

1. Gate the verbs include in `tcp_transport.h` behind `#ifdef` (or move verbs-dependent bits to an rdma-specific header) so the TCP transport compiles where only a plain socket stack exists.
2. Same for `` in `common.h` (fallback stubs when `__APPLE__`).
3. Relax the `setup.py` darwin hard-exit into an explicit `--allow-unsupported` / experimental wheel path, since the tag helpers are already written.

If maintainers are open to it, I can start with (1) + (2) as a small PR targeting a TCP-only `USE_TCP=ON`-style CMake option for non-Linux platforms. Context: we're wiring PD disaggregation across a pure Mac cluster (vllm-metal#709) and Mooncake's TCP backend is the obvious production-grade transfer layer to grow into.

Contributor guide

Open the contributing guide

Research direction

Start with mooncake-transfer-engine/include/transport/tcp_transport/tcp_transport.h and common.h, then inspect mooncake-wheel/setup.py and the CMake configuration. Try a TCP-only transfer-engine configure/build on macOS and check the wheel install path. Done means the TCP path no longer requires verbs or numa headers and the darwin guard no longer blocks the proposed experimental build.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, macos, python
Domain
build-system, networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.